The Hardest Lesson

A year ago I experienced one of the hardest lessons so far in my early career. It was impactful in so many ways that it led me to create this post. It was impactful because it led me to question so many things and even led me to take a mental day off of work. Yes you heard that right. I had to take a day off of work because mentally I was just not feeling well. I'm very grateful that one of my former managers allowed me to do this despite what happened at the very end months later. To me, a lesson should be something that not only we learn from but motivates us to continue to hold firm our values even when people disagree with them! In other words, it's continuing to be our authentic selves and respectfully challenging others that the problem is not about thinking differently, the problem is not respecting those who think differently. I hope that this post resonates with others that have also experienced such a hard lesson before in their career.

The Beginning of a Good Crisis

It started on May 13th 2021, I was off call that week working on infrastructure related projects. I was also still in grad school and final exam week was approaching. It seemed like any other week really. For example, whoever was on call for IT support had weeks where it was slow and others where it was busy. When it was slow, sometimes if someone who was off call needed help, then the person on call would jump in to assist. I rarely asked for help during off call as I did not like to bother people and instead I would do a lot of research on my own to solve issues. When I had completed my tasks during off call, I liked to do the same for others and ask if the person on call needed help. I also liked to keep an extra eye on my team's Slack channel to see what Jira tickets were coming in. The beginning of a good crisis started just like this when I saw a concerning ticket come in through the team Slack channel.

A Persistent Threat

I saw a ticket from someone on the customer success team notifying that they had received a phishing email from a customer of theirs. The phishing email had the same subject of the phishing email that an employee on the finance team had fallen victim to whom had entered in their credentials on a credential harvesting page designed to look like an Office 365 sign in page. The person on the finance team had reached out to me via Slack. This is something that still irks me till this day because I could have been on PTO or sick. I found that at times people would just directly Slack me especially if it was something security related even though I was not on the information security team. The protocol was to email either the IT team email alias so that not only a ticket is created, but also everyone on my team is informed, or an email to a suspicious emails handle that the information security team oversees. However I would find out much later that not following protocol came to be of some use since the affected person's email account was already compromised, as I began to investigate and learn more details about the security incident. Right away, I stopped doing my off call work and alerted my coworker and manager of what was going on. We disabled the affected user's Office 365 account, reset their password, and also escalated the incident to the information security team, as we continued to gather more evidence to see who else could have been affected to really see the bigger picture or "blast radius" or scale of potentially more compromised users. This is where I came in the day later.

Investigating the Blast Radius

It was a day later around 3:00am. Determined and persistent to find more evidence to see if the blast radius was wide, I stayed up late for several reasons:

• I was devoutly interested in the problem, and when someone is that interested in something, they give it their all no matter if someone told the person to take action or in this case stay up late to gather more evidence.
• Having the initiative to do something when nobody tells them anything, is what differentiates between someone who is just comfortable coasting, and someone who is relentlessly curious in seeking more answers.
• Last but not least is doing the right thing. As cliché that may sound, not many people do the right thing and would rather just let other people do it. These people are the type of that say, "If it's not anything I usually do, then I don't have to do it and would rather let someone else handle the work."

I would say that the most important point to take out of here is that when you're working on something where you have more questions than answers, you should probably really take some time to think and ask yourself why there are so many questions than answers in the first place. This is why curiosity is the greatest characteristic anyone can have. You're always seeking answers and persistent in finding them! Every InfoSec professional out there would agree with me on this because it's something that is both valued and vital to have in this field! Now that we got this out of the way, let's uncover what was found.

Azure AD Sign-ins Ground Zero - Emerging Details

I began to look at the Azure AD sign-ins logs. I filtered the search criteria with the subject of the phishing email, and sent emails from the compromised user. With just these two filters, I came across more evidence. The scale of compromised users was much more. There were about 40 compromised users which the phishing email was shared by the person on the finance team unaware that someone had gained access to the person's account and was sending such emails. This was before the compromised user's account was disabled and password later reset based on the date and time. The source IP address of the successful sign ins were coming from Boardman, Oregon. I knew that it was impossible for so many people to be at that location let alone in a pandemic. This was clearly malicious and even matched the same location from the initial compromised user. Given how there were successful sign-ins coming from so many people's Office 365 accounts, I was now on high alert and decided to end my investigation that early morning and be the bearer of bad news informing both my team and InfoSec that the blast radius was more wider than expected, and we should disable and force reset everyone's password, and enable MFA as there may be more compromised users that we are unaware of.

On High Alert Teams Assemble!

I slept a few hours and somehow I felt well rested. Not sure if it had anything to do with the point I made earlier about being devoutly interested in something that somehow even with a few hours of sleep it felt like I slept many hours. Both the director of InfoSec and VP of infrastructure had acknowledged my evidence and recommendation to force reset everyone's password. It was a moment to feel good about myself considering how I managed to support my reasoning during the very early morning hours, as well as convince higher up that we needed to assemble and work together considering the severity of the situation at hand. I began to answer any questions the InfoSec team had and communicated back and forth via a newly created Slack channel designed solely to communicate about the incident as well as report any additional findings including ones that had already been made. Since the major impact was to confidentiality since there was unauthorized access to information and such information may have been disclosed already publicly, we reached out to the CTO as well as legal to see who else needed to be involved to help out. We began to reach out to more teams such as HR/people ops, rev ops, customer success, and eventually team CEO. We also notified the UK's Information Commissioner's Office (ICO). There were many teams now working on the incident. In a period of two weeks, everyone worked together meeting up on Zoom to better understand the consequences at what had happened. It was surely difficult for everyone. So many hours clocked in that I never had clocked in before. Not only this but final exams week was approaching and I had deadlines to submit assignments/projects, present them, and take actual exams. It was very stressful for everyone. However something that helped cope with the workload and stress was from the work environment itself. The startup environment was very fast paced and priorities changed constantly. Not only that but so much had happened before the incident that I felt prepared in a way is the best way to describe how I felt. This helped me become much more resilient something that is valuable to have. As the saying goes, "when the going gets tough, the tough get going".

Post Mortem

A simple post mortem was done to ensure that we're all held accountable for what had happened, but also be proactive to ensure such incident does not occur again and if it does, the damage is minimal because we're more prepared. Nobody did any finger pointing and most importantly we worked as a team and acknowledged that we simply needed to do better. A google doc was created detailing the timeline of events, followed by a section of the actions taken with screenshots included. Many people who's Office 365 account was disabled and password reset were either sick or on PTO which was a bit frustrating because we never ended up stopping our main line of work which was providing IT support. The CTO had to inform everyone that support was going to be slow due to my team's involvement with the incident. So the massive workload was definitely a challenge but to me it was welcoming as I said before, I have grown to be more resilient. With the post mortem finished, eventually everyone was informed of what happened in a meeting. Lastly MFA was enabled and enforced across the board. Why was MFA not enabled in the first place?! This is something that I unfortunately have very little details on, and so it does not make sense to talk about something I know very little of. Would having MFA enabled and enforced prevented the incident? Most likely yes especially when used with an authenticator app and not SMS due to SIM jacking. However in security, there is no such thing as something perfectly secure. This is why we need to be vigilant and not let our guard down. Upon having enabled and enforced MFA for everyone, a month later the HR team had announced an annual performance review now with the option to pitch for a role that does not exist.

The Perfect Moment

I attended a meeting that the HR team held for those interested in knowing more about the process in pitching for a new role that currently did not exist. I asked questions about how the process works. It was very simple. The process was as follows:

• You pitch the role to the head of the team. In this case I was pitching for an IT Security Engineer role and the head or who would become my future manager was the director of information security. My current manager would also be present in the pitch.
• If the pitch was well received from the head of the team and those in attendance, then the head would speak to higher up in this case HR management to see if there is actually a need for such role at the company, and from there allocate budget by removing another role.

The pitch I presented was very well received. I got help preparing from a former manager and a former coworker who showed me their pitch in the form of a deck. The key reasons why I mentioned such role needed to exist were the following:

• There was a lot of friction with my team and InfoSec. For example, miscommunication which resulted in a lot of back and forth or "she said he said" type of discussions. This is something I was strongly against as I was experiencing this with other teams as well. I was very concerned that this continued to happen and nobody would say anything or speak up. This of course creates risk which is added on to the risks that already are known. Why create more risk? This is what was going on through my mind which I told myself it needed to be handled more efficiently.
• The company had just experienced an incident nobody was prepared for. We must learn from it by investing resources and or potentially adjusting where we are spending budget on. Having a dedicated IT Security Engineer role where someone who has more familiarity with Windows infrastructure and has been working at the location for quite some time and knows the systems well, is very beneficial. Most importantly is having an intermediary to ensure teams are working together to better understand each other's priorities which is also something that was constantly a struggle "this is not high priority for me but for the other team it is!" type of discussions. A person who is able to facilitate such conversations so that teams are aligned and priorities are clear, can be the difference between a healthy working relationship where we have each other's backs and I trust you, to I don't like working with you and I don't trust you which could result in an another possible security incident because of the added risk of teams not working together because there is no trust.
• Helping with security questionnaires because the company was growing considerably as well as creating security whitepapers to show future customers that the company is reputable in security because it follows policies and procedures established by an information security standard such as ISO 27001.
• Last but not least is how I had recently obtained my master's degree in Cybersecurity. I had already expressed my interest long ago when I started my employment that I was interested in InfoSec and already had some personal knowledge and technical experience with ethical hacking obtained from an internship. I was also one of the first involved at ground zero and alerted my team and InfoSec about the emerging details I uncovered in the early morning hours. If I had not done this, the situation would have been much more worse where the phishing email would have reached more users, which could have resulted in more compromised users leading to more unauthorized access to information that would have been exposed and be leaked to the public. Having more eyes and manpower definitely is crucial here.

These were the main reasons why I pitched. I took the time to support my reasonings, by providing clear concise evidence of what I had witnessed and gained in experience. Three weeks later after pitching, I received unexpected news.

Unexpected News

What seemed like the perfect opportunity for me to grow in my career and finally solve all of the problems I wanted to solve I mentioned in my pitch and most importantly do something I am passionate about and love every freaking day was not going to happen. A former manager had informed me that they had bad news and good news for me. The bad news was that I was not going to get the role because there was no budget for it. While the good news was that I was being promoted and the promotion came with a $8,000 raise. At the moment I heard about the bad news of not getting the role, I felt like someone had ripped my soul from my body. I was no longer in my body anymore and the good news along with everything else did not matter to me. I remember having a lot of difficulty comprehending the rest of the discussion with that manager. It became difficult for me to continue listening pretty much. Some people may label me as exaggerated or ungrateful and I respect everyone's opinion. The raise was definitely nice to get. Ultimately however I was not happy. I asked my manager, who was the person that made the final decision in all of this? I wanted to know if there was anything that could have been done to change the decision, and they let me know that there was nothing that could be further done. Much later after work the same day, I began to ask myself so many questions mainly on how could there be no budget given what had happened. Is reputation and customer trust not important? I could not stop thinking of what I had heard earlier. When I went to sleep that same day, I had a lot of trouble falling asleep that I took a day off of work because mentally I was not feeling well.

Learning a Hard Lesson Takes Time

In the year since that event, I was eventually let go because I was no longer a good fit. I received six months of severance pay, and I decided I just needed to take some time off to not only recharge, but think of something else that is not security related which as you know in a previous blog post, is pretty difficult for me! Since that day when I heard there was no budget for the role, I had continued working on what's called a financial lateral move, which I would find out later was not the best for me in terms of career growth. Not only that, but some of the problems/reasons why I wanted the role that it was going to tackle started to occur more frequently, and this right here was a clear indication that I had reached my end. I learned so much during my time that it may seem ignorant for me to not say so! I also made great connections and some may even say friends, who reached out to me even outside of work hours to just randomly check up on me. To those people and you know who you are, thank you so much for being empathetic and caring for my mental well-being. I also finally understood what someone once told me who later left the company that is was going to be difficult to get the role. They informed me of something obvious I knew already which was the pay difference with talent in the US and talent overseas. However to me it's as if that person was letting me know of something else which I must have overlooked. I am also very happy that many people out there have similar sentiment on how important it is to not only invest in security, but also make it part of the overall culture of the company. From the lowest person on the totem pole, all the way to the owner of the company. A key takeaway from this very interesting reading from Dark Reading titled "Security Stuff Happens: What Do You Do When It Hits the Fan?" has to deal with communicating gaps. It says, "Organizations experiencing a security incident must not hide behind a third party and shouldn't blame their employees. They also must not allow lawyers to create smokescreens around what happened. This helps no one in the long term and only saves face until it doesn't anymore." I am not aware of this happening at all but this is definitely worth addressing because this is an example of poor security culture because it is untrustworthy, deceitful with no integrity when you try to cover something up! The culture is key section also is very interesting to me. It says, "No one with a straight face can deny the importance of security culture when it comes to keeping a tight ship, managing third-party security, and mastering internal communication. Culture weaves through it all. Motivated employees with excellent support from their team members and leadership are less likely to make errors and are also less likely to turn around and give information to a cybercriminal when faced with the temptation of shiny rewards or, worse, revenge." A tight ship is definitely important because when your employees are motivated, they aren't just there for their paycheck. They are motivated because they are supported and the problem they are facing is important to them. Security incident response goes far beyond technology. It's a, "pressure test of culture and security leadership. Companies with good culture — one that embodies true collaboration and solidarity — are much more likely to suffer minimal internal damage after a security incident".

The Path Ahead

This post as I mentioned in the beginning was something that I experienced and it was impactful to me. I now know that the hardest lessons we experience in life are those that even when we don't get what we wanted, we still stay true to ourselves and persevere till we get what we want. As I search for my next opportunity, I already know what I want from it. All the growth I have experienced, I cherish it so much because it's what's gotten me to where I am at right now. I feel much more prepared now and ready to take on this world that although can be unfair to many, one can NEVER stop believing in what they truly love, and for me that is making a difference for people through security. I've learned that being too passionate about something is risky because not everyone is just as passionate as you are. You cannot have a tunnel vision on what you believe in because again, not everyone has that passion and drive qualities in them in regards to security. This to me was my hardest lesson. To end, this quote from Winston Churchill used by other InfoSec professionals is something to keep close to, "Never let a good crisis go to waste".