a permissions problem. You say that there's an empty directory inside the one you want to delete - have you tried deleting that first? Most FTP applications will by default refuse to delete directories that have anything in them. You can usually change that default to allow the deletion of non-empty folders, but I don't use FTP Voyager, and so I'm not sure if you can do it/how you do it.
Does the server allow you to rename or move folders?