Debugging and Customer Support stories to share?

Discussions
#1

I like to produce some quality content for this service-based business. The service is High-Tech Customer Support and Bug Fixing (mainly for GUI-heavy software products).

To come up with ideas for content I like to ask everyone whether you have any stories or experiences about customer support or bug fixing that you like to share.

What you learnt from the experience, what was particularily difficult about the support case or the bug, what tools or techniques you used, what was the outcome whether positive or negative and what you would do if you were in that situation in hindsight?

#2

I remember someone signing up to Hexadecimal and adding 19 websites in a very short time. One of them was marked down straight ahead. What threw me off was that the website was accessible from the browser but not the command line (unlike anything I saw before). So something along these lines:

curl -v https://incomplete-chain.badssl.com

curl: (60) SSL certificate problem: unable to get local issuer certificate
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it.

After trawling through various mailing lists for about 2 hours and poking around the command line, I found out that their web server was misconfigured such that it had a missing (intermediate) certificate in the chain. Browsers can work around this by using the Authority Information Access extension (AIA) to download missing certificates in the chain, while curl (and OpenSSL) can’t do that. I had to manually add the missing certificate to all my servers and update the certificate store before the problem was gone.

Soon after I found about the issue, I shoot them an email letting them know that I’m working on it. After I solved it, I prepared a short postmortem describing what went wrong.

1 Like
#3

Interesting story @jmstfv. Thanks for sharing :slight_smile:

I happen to have worked and be working with a client at visualSilicon on this very subject of certificate chains and digital signature validation and I happen to have this problem of missing certificates in the chain because actually the product in this case is supposed to verify when it is offline and to come up with a positive user experience while keeping the security intact is hard. These stuff sound simple on the surface but get very complicated very soon.

One question I would ask in your case is, did you eventually implement the method browsers implement (downloading missing intermediary certificates to complete the chain automatically)?

#4

This is something that I’d love to tackle in the near future, but I’m not keen on implementing the AIA extension myself. There are two ways to work around it:

  • Periodically download intermediate certificates and update the certificate store on all servers. This is more of a brute-force approach, and it will probably work if I could get my hands on those certificates.

  • Running a headless browser. Why reinvent the wheel if the Chromium & friends has already implemented it? On top of that, running a browser has benefits such as advanced rendering engine (unlike curl) and accurate request timings. The downside is that it is operationally harder and more expensive to run a fleet of headless browsers than a fleet of Sidekiq workers performing curl requests. I’d think thrice before introducing such complexity.

Oh, I have the same sentiment about a lot of things on the Internets. SMTP, IMAP, SSL/TLS, OpenVPN, rendering engines (e.g. Gecko, Webkit). Heck, something trivial like favicons quickly gets out of hand once you start factoring in various devices and operating systems.

1 Like