topic Re: Who or what is Little Hammer? in Administration and Deployment

Who or what is Little Hammer?

jklaverstijn — Thu, 16 Aug 2018 12:25:35 GMT

We are running quite a few metadata clusters and have our fair share of dealings with that. Mostly they are discussed with SAS Technical Support but this one stays obscure. Maybe you have come across the following?

Every now and then, usually in the context of more serious problems (loss of quorum for that node), we find Little Hammer in the metadata server log. And I got slightly amused and very curious after the backgrond of this. Mind you its logged at the INFO level so you will probably not need a lot of trace/debug logging to come across it.

2018-08-16T13:32:07,998 INFO [00001773] 5:sas - Waiting for Little Hammer to be done
2018-08-16T13:32:13,405 INFO [00001773] 5:sas - Little Hammer is done

Has anyone else seen this? And do you have any clue as to what or who Little Hammer is and what he/she/it does? And does Little Hammer have a big mum and dad?

Looking forward to your insights!

Regards,

-- Jan.

Re: Who or what is Little Hammer?

alexal — Thu, 16 Aug 2018 12:43:09 GMT

@jklaverstijn,

When Metadata Servers in a cluster come up, they need to determine the most recent journal entries available and make sure all nodes are up to the same update, presumably insuring all metadata is identical between all nodes, before they can begin responding to metadata requests from clients. The "Little Hammer" determines that slaves are at least as recent as the current backup and just need journal entries from master's journal file. Those entries needed by each slave are submitted to those slaves and all slaves are queried at the end to make sure they are now all up to date.

When Metadata Servers in a cluster come up, occasionally they will find that one of the servers has been out of commission for quite a while and houses metadata that's way out of date with the rest of the cluster, such that it cannot be brought up to date by simply feeding the server journal entries. In that case, the "Big Hammer" is to simply request the server to recover itself from the master server's most recent backup which will put it at the point where the master server's journal started. Once recovered, the master server will re-evaluate the slave and discover that it can be brought up to date by running the "Little Hammer."

Re: Who or what is Little Hammer?

jklaverstijn — Thu, 16 Aug 2018 13:26:51 GMT

Hi @alexal,

Many thanks for this insightful explanation. Very educating. And certainly reassuring as I am conditioned to being afraid of the hammer, little or big. Now that I understand what's happening this all changes.

Certainly there exists a secret society where all this can be learned. 😉

Regards,

-- Jan.

Re: Who or what is Little Hammer?

alexal — Thu, 16 Aug 2018 13:28:44 GMT

@jklaverstijn,

You are welcome.

Re: Who or what is Little Hammer?

JuanS_OCS — Thu, 16 Aug 2018 13:38:35 GMT

Hello @jklaverstijn, @alexal,

thank you! I did not know about this either. 🙂