The problem with certifications

Certifications sound like a great idea, and if I were in HR recruiting IT people, I could be forgiven for thinking that they tell me something important about a person’s skills level. But I would be wrong.

The idea of certifications is not wrong. If I were having a boiler installed, I would probably want the technician to be certified to work with gas. This would be an industry certification, perhaps with an independent assessment body. They might also have done some training for the specific type of boilers they install. The IT industry does not do this. It has vendor certifications that are intended to demonstrate a skill level with a specific technology:

  • Cisco Certified Internetwork Expert (CCIE) and Cisco Certified Design Expert (CCDE)
  • Microsoft Certified Solutions Expert (MCSE)
  • VMware Certified Design Expert (VCDX)

The certifications are hard. Anyone who has a CCIE certification has demonstrated the ability to study and has the aptitude to pass a certain type of exam. They may also have been fortunate to have an employer willing to pay the steep fees for courses. The question, however, is whether they demonstrate real expertise in the technology.

The problem is that the course material is created with the idea of enabling an exam, and the exam is created purely as a test of the course material. An example will show what I mean.

This example is taken from the Cisco material for TSHOOT. This is one of the exams for the Cisco Certified Network Professional (CCNP) Routing and Switching certification. It covers the skills for troubleshooting and maintaining Cisco IP Networks. Cisco certifications are some of the best, so this example is not an adverse comment on Cisco. It is just an example of a certification topic.

Troubleshooting an IP network requires a good understanding of TCP/IP, and how packets flow through a network from server to switch to WAN and client, and back to the server. NetFlow is a way of recording information about flows, so that you can diagnose performance problems. There is quite a lot you need to know about flows in order to diagnose problems. The course material tells us that:

"A flow is a unidirectional stream of packets, between a given source and a destination, that have several components in common. The seven fields that need to match for packets to be considered part of the same flow are as follows:

  • Source IP Address
  • Destination IP Address
  • Source Port (protocol dependent)
  • Destination Port (protocol dependent)
  • Protocol (Layer 3 or 4)
  • Type of Service (ToS) Value (differentiated services code point [DSCP])
  • Input interface."

I suppose there are a number of concepts here. One is that the flow is a specific "conversation" between client and server. Now this is a bit surprising. It says "unidirectional". Does that mean that the response to a request is in a different flow? How can I tell if there is a network or server delay if the request and response are in different flows? Another concept is that you can’t jump between interfaces. You might have more than one network connection to a switch, but those would be separate flows. I don’t really need to know that there are precisely seven fields: I can always look that up. And I don’t need to know trick questions about what might be a field but is not. TCP/IP flows is a really interesting topic, and I would like to know a bit more about it.

Now here is the test question:

"Which of the following is not a NetFlow key field

  • Source IP Address
  • Layer 4 Source Port
  • ToS Byte (DSCP)
  • TTL
  • Input Interface."

Did you notice what happened there? I don’t need to know anything about flows. I just need to remember the list of seven fields. And I need to be aware of trick answers. Is Source Port really Layer 4? Is TOS value really a byte? Did I just forget TTL, or could there be a reason why Time to Live is a field that I have forgotten? None of this matters in the real world. In the real world we switch on NetFlow, and configure a collector like SolarWinds. The real work is in interpreting the NetFlow data. And NetFlow is expensive. And it can’t tell you what is happening on the parts of the WAN you do not control. And it does not tell you what response time the user actually experiences.

The problem here is the methodology. If the exam were in Algebra, there would be a vast body of knowledge, different course material, trained teachers and professional examiners. But there is no such body of knowledge or of educators for troubleshooting an IP network. Cisco has to get someone to prepare a course and someone else to prepare an exam. The exam is a test of the course.

Certification courses provide useful training. And simple exams that test whether you paid attention are OK. But certifications do not prove skills. They prove a willingness to study, and an aptitude for a certain kind of test.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.