Some of us will at times have played with someone who has tried to call us up for the action of brushing something off our line on the putting green with our hand, saying something along the lines of, “You can only do that with a towel / glove / club but not your hand.”
So, who is right? Thankfully, not them, as long as we are taking "brush the green with your hand" to mean brushing away sand or loose impediments rather than rubbing the surface to test the putting green of the hole you are playing. That latter act of deliberate testing would get you the general penalty under Rule 13.1e, as would rolling a ball along that putting green to deliberately test the surface before you had finished playing that hole. However, both such actions are permitted on the putting green just completed between the play of two holes.
But assuming this is not what you are doing, then brushing the green with your hand to remove sand or loose impediments from your line is absolutely fine.
Under the Rules governing the putting green, 13.1c says that you may remove sand or loose soil on the putting green without penalty, but without stipulating how that should be done, so you may use a towel, a glove, a club or your hand, as you wish.
When it comes to loose impediments such as stones, twigs, leaves etc, Rule 15.1a does give a little more detail, stating that, “Without penalty, a player may remove a loose impediment anywhere on or off the course, and may do so in any way (such as by using a hand or foot, using a club or other equipment, getting help from others or breaking off part of a loose impediment).”
The part of Rule 13.1c relating to sand and loose soil also says that you can’t remove them anywhere else on the course without penalty, only on the putting green. While we are talking specifically about the green in our headline question, it’s worth mentioning that “anywhere else on the course” includes the apron or fringe around the putting green. Neither of these two words are used specifically in the Rules of Golf, but both form part of the general area.
Neither sand nor loose soil are classed as loose impediments in the Definitions section of the Rules, so Rule 15.1a (quoted above) does not apply to them. This means that if your ball is lying off the green and there is sand on your line on both the fringe and the putting surface, you may brush it away on the green with your hand, towel or whatever, but may not touch the sand on the fringe unless, for example, it was deposited there by another player’s bunker shot after your ball had come to rest.
In that scenario, when the conditions affecting your stroke have been worsened by another player, you would be entitled to restore the conditions that prevailed when your ball came to rest under Rule 8.1d.
But going back to the main question here, the simple answer is ‘yes’, as long as you are not rubbing the putting green you are playing with your hand to deliberately test the surface.