BookmarkSubscribeRSS Feed
🔒 This topic is solved and locked. Need further help from the community? Please sign in and ask a new question.
jklaverstijn
Rhodochrosite | Level 12

We are running quite a few metadata clusters and have our fair share of dealings with that. Mostly they are discussed with SAS Technical Support but this one stays obscure. Maybe you have come across the following?

 

Every now and then, usually in the context of more serious problems (loss of quorum for that node), we find Little Hammer in the metadata server log. And I got slightly amused and very curious after the backgrond of this. Mind you its logged at the INFO level so you will probably not need a lot of trace/debug logging to come across it.

 

2018-08-16T13:32:07,998 INFO [00001773] 5:sas - Waiting for Little Hammer to be done
2018-08-16T13:32:13,405 INFO [00001773] 5:sas - Little Hammer is done

Has anyone else seen this? And do you have any clue as to what or who Little Hammer is and what he/she/it does? And does Little Hammer have a big mum and dad?

 

Looking forward to your insights!

 

Regards,

-- Jan.

1 ACCEPTED SOLUTION

Accepted Solutions
alexal
SAS Employee

@jklaverstijn,

 

When Metadata Servers in a cluster come up, they need to determine the most recent journal entries available and make sure all nodes are up to the same update, presumably insuring all metadata is identical between all nodes, before they can begin responding to metadata requests from clients. The "Little Hammer" determines that slaves are at least as recent as the current backup and just need journal entries from master's journal file. Those entries needed by each slave are submitted to those slaves and all slaves are queried at the end to make sure they are now all up to date.

 

When Metadata Servers in a cluster come up, occasionally they will find that one of the servers has been out of commission for quite a while and houses metadata that's way out of date with the rest of the cluster, such that it cannot be brought up to date by simply feeding the server journal entries. In that case, the "Big Hammer" is to simply request the server to recover itself from the master server's most recent backup which will put it at the point where the master server's journal started. Once recovered, the master server will re-evaluate the slave and discover that it can be brought up to date by running the "Little Hammer."

View solution in original post

4 REPLIES 4
alexal
SAS Employee

@jklaverstijn,

 

When Metadata Servers in a cluster come up, they need to determine the most recent journal entries available and make sure all nodes are up to the same update, presumably insuring all metadata is identical between all nodes, before they can begin responding to metadata requests from clients. The "Little Hammer" determines that slaves are at least as recent as the current backup and just need journal entries from master's journal file. Those entries needed by each slave are submitted to those slaves and all slaves are queried at the end to make sure they are now all up to date.

 

When Metadata Servers in a cluster come up, occasionally they will find that one of the servers has been out of commission for quite a while and houses metadata that's way out of date with the rest of the cluster, such that it cannot be brought up to date by simply feeding the server journal entries. In that case, the "Big Hammer" is to simply request the server to recover itself from the master server's most recent backup which will put it at the point where the master server's journal started. Once recovered, the master server will re-evaluate the slave and discover that it can be brought up to date by running the "Little Hammer."

jklaverstijn
Rhodochrosite | Level 12

Hi @alexal,

 

Many thanks for this insightful explanation. Very educating. And certainly reassuring as I am conditioned to being afraid of the hammer, little or big. Now that I understand what's happening this all changes.

 

Certainly there exists a secret society where all this can be learned. 😉

 

Regards,

-- Jan.

 

 

alexal
SAS Employee

@jklaverstijn,

 

You are welcome.

JuanS_OCS
Amethyst | Level 16

Hello @jklaverstijn@alexal,

 

thank you! I did not know about this either. 🙂

suga badge.PNGThe SAS Users Group for Administrators (SUGA) is open to all SAS administrators and architects who install, update, manage or maintain a SAS deployment. 

Join SUGA 

CLI in SAS Viya

Learn how to install the SAS Viya CLI and a few commands you may find useful in this video by SAS’ Darrell Barton.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 4 replies
  • 1130 views
  • 7 likes
  • 3 in conversation