The SolarWinds hack, how does it affect you?

Background

As if this year has not brought enough turmoil and chaos to the country and world, this hack shows that 2020 had at least one more trick up its sleeve. In case you have not heard already attackers were successful in inserting a trojan into a widely used enterprise IT management called Orion from the software company SolarWinds. We may never know the full scope of this hack but any company that installed or updated Orion in the last 6-9 months is potentially at risk. It is very likely before this week you have never heard of Solar Winds so how bad can this be? Solar Winds has a large customer base as shown by the screen capture below from their now removed webpage.

How Widespread is it?

It is estimated over 18,000 entities used SolarWinds Orion product suite. Over the next days, weeks, and months you will probably hear of more and more companies and government bodies that have been breached. This hack has affected local governments from Pima County to highly sensitive Federal government agencies such as the Department of Energy’s National Nuclear Security Administration as well as the U.S. Treasury and Commerce departments. Several organizations such as Cox Communications, Microsoft, Ford, and many others have been affected.

How bad is it?

The attackers were not out to cause chaos and destruction, that would have been a waste of their efforts that would have brought this hack to light very quickly limiting its effectiveness. Rather they built some intelligence into the trojan such as not executing until several weeks after installed. They took efforts to make sure the files were signed and trusted like all other SolarWinds software. The trojan’s communication with the hackers was disguised to look like the SolarWinds traffic in order to obfuscate it and prevent detection. This hack was probably months or years in the making and there is currently no information on how it was accomplished. Some of the likely ways were; directly hacking into SolarWinds systems, using reused credentials, or paid an employee to help with the attack. It was most likely a nation-state actor such as agents of the Russian government from the current indicators.

Is my company at risk?

From what is currently known, only companies that had SolarWinds Orion with specific versions are affected. Orion is a product that is typically only used by larger companies as it is very expensive and is used to manage large computer networks. If you did not have this product installed on your network you are not at direct risk from this attack, most likely you did not. Any company that does or did have this product installed in the last year should immediately take steps to investigate, remediate and recover. Luckily Microsoft and others have taken steps to take down the command-and-control network used by the attackers, but that does not mean they did not get alternate access to breached networks and time is of the essence in situations like this.

Even without the product installed you and your organization can still be at risk. The highest risk will be from your information being used by the attackers. Being they accessed companies, ISP’s, hospitals and government agencies any sensitive information they had could be out in the wild. There is little to no direct risk to you or your organization from being hacked through any of these vectors. Except for in very care cases, your ISP such a Cox Communications being breached, did not and does not give the attackers inside access into your networks. Most businesses use a router/firewall which is not managed by Cox or other ISP’s and would make their network being breached no different than any other internet attack against your company.

With that said the attackers wanted high value targets and would not have had the resources or want to risk their efforts being caught by going after what would be a low value target to them. Government agencies, large companies, defense contractors, hospitals, Universities and ISP’s are their most valuable targets.

What now?

If your organization uses or recently used Orion, I hope you are not still reading this and have already contacted your IT department, Lawyers and a good cybersecurity incident response company such as Cynet, FireEye (yes that is another story in itself), CrowdStrike, SecureWorks or others.

For the rest of us the best thing we can do is take steps to increase our security posture. It is not possible to stop all threats but good cybersecurity hygiene and best practices will protect us from most attacks reducing the likelihood of a breach. Many organizations already have compliance standards they might need to follow based on their industry, such as HIPAA for health care providers, PCI for retail businesses, NIST 800-171 for Department of Defense contractors, SOX for publicly traded companies, Gramm-Leach-Bliley Act for financial institutions, just to name a few.

For companies that do not fall in any of the mentioned regulations, the NIST Cybersecurity Framework is a good starting point for processes and procedures to harden and company against cyber-attacks.  This framework is a scaled down version of the NIST SP800-53 which is cost prohibitive for most small companies. The CSF has guidelines to help a company Identify, Protect, Detect, Respond and Recover as shown in the graphic below.

Final Thoughts

Cybersecurity is more than just implementing technological safeguards. It is technology along with ongoing administration and processes with the goal of reducing risk. This will require some internal change and, in most cases have financial costs associated with it. The cost of not taking steps to protect the organization could be much higher in the terms of reputation, loss of business, government fines or lawsuits.

Java issues administering HP printers and network equipment

With newer versions of Java and older firmware on HP equipment you may run into Java errors trying to administer the devices through the web interface.  Adding the IP or FDQN to the Java exceptions alone will not work now that High security is the lowest level allowed in the Java control panel. Below are a few fixes I have found.

For an HP ProCurve 2810 switch there needs to be an additional entry in the java control panel in the form of http://x.x.x.x/classes/agent.jar

2016-06-17 23_36_10-ScreenConnect - CONTROLLER-PC - Connected

For an HP 4650 printer I found the java.policy file needs a line to allow the specific port used by the printer interface, this case 161, that is not a normal HTTP/HTTPS port. The file was located in “C:\Program Files (x86)\Java\jre1.8.0_77\lib\security” which could be different based on your JAVA version. A line similar to ‘permission java.net.SocketPermission “x.x.x.x:port#”, “connect, accept”;’ will need to be put in the grant section of the file. This is in addition to the http://x.x.x.x in the Java control panel

2016-06-17 23_37_49-C__Program Files_Java_jre1.8.0_91_lib_security_java.policy - Notepad++ [Administ

I hope this helps some other people that run into the same issues. If you have any other fixes I would love to hear them.

QuickBooks 2015/2016 Unrecoverable Error

As any IT person can probably attest to QuickBooks can be a challenge sometimes, it is a widely popular accounting package used by  millions. Most issues arise when it is used in a multi user environment, from simple issues like firewall ports for the DB server, user error like leaving it in single user mode or making their workstation the host, to crazy issues like QB no longer liking the file name and and the only solution is to rename the file. Well this time it turned into tens of hours of frustration and talking to the Office of the President….

Haha, this is only at Intuit we are talking about, not the White House, now back to the problem.  The issue first came up when trying to move from QuickBooks Online to QuickBooks Desktop, every time the export process would fail. I called QBO (QuickBooks Online) support, they were very kind and helpful yet after about an hour of trying still the same result. Their next step was to send me a QB portable file that they created, later that day I got the file, restored the file and QB crashed with an unrecoverable error. Tried a few things with no change, so I decided to try another computer. Well same result once again, another unrecoverable error. Was there a problem with the QB file or was this a bigger issue?  I tried creating a new company file and also opening a sample file, yet again the dreaded unrecoverable error.

2015-10-14 08_42_47-qbunrecoverableerror.png - Windows Photo Viewer

Doing some research it appeared to be a .Net related issue, I uninstalled and reinstalled .Net and repaired QB installation. No change, still getting the error.  I installed QB on another 3 computers and they all had the same issue. I uninstalled some programs that were the same between computers and still no change. Not wanting to get too bogged down by this issue and hoping their support could use the QB error logs, I thought I would call QB support. After an hour or so of working with Tier 1 support I asked to be escalated to Tier 2. In all I worked with 4 different people over about 4 days with Tier 2 support, then with a Supervisor. Quickbooks 2016 came out during this and still had the same issue. Ultimately the supervisor told me to reload Windows on all of the machines. I was not happy with this solution so I emailed the VP of Small Business at Intuit who is the head boss over QuickBooks.

Within an hour I got an email that the Office of the President at Intuit would contact me the next business day. The next day I got a call from someone in the local Tucson Intuit location. I rehashed everything that had been gone through and we exchanged some emails. I was then told that they would send someone onsite. Now a week later a person from the local site shows up, and we try many of the things that had been done with Tier 2 support. At this point I have blood running down my forehead from banging my head on the wall at the absurdity of doing the same steps that did not work before over and over again. In talking with this person I glean that while I had sent many of these error reports in that nobody had even looked at any of them. The next step was to have development look at the error logs that have been going into limbo at Intuit somewhere. So now I wait again.

After about 5 days I get a email that they have learned something from the logs and have a fix they would like to attempt, so I arrange to do a remote support session in the morning. It turns out one of the 3rd party programs on my suspect list, IBM iAccess 7.1, might be the culprit even though uninstalling it did not fix the issue. This program adds a bad entry into the machine.config file for .Net off to edit the file at ‘C:\Windows\Microsoft.NET\Framework\v4.0.30319\Config\machine.config’ and ‘C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Config\machine.config’

iaccessQBissue2

The iAccess software had added an extra line to the machine.config file which is XML and needs to be properly structured. Once the <DbProviderFactories /> line was removed from the machine.config files we attempt to open Quickbooks and it now works. I tried this on the rest of the machines and they all work now. Lesson learned on this one, delete all files even though .Net was uninstalled. I am grateful that Intuit cared  enough to stick through this and help find the resolution rather than just insist on reloading Windows on all of the machines. I am also investigating this support article from IBM to see if it is related to the problem http://www-912.ibm.com/n_dir/nas4apar.nsf/ALLAPARS/SE45767

Cotap Targeted Spam

Cotap, a company that offers ‘secure’ team messaging for businesses has gone an unethical route and is using targeted SPAM emails to solicit customers. You get an email saying you have been added as a contact for the company, but oddly enough nobody in the organization has used or heard of Cotap’s service.  From searching the internet and seeing posts in the forums at Spiceworks it looks like this service has been doing this for a few months. I would suggest that you mark the emails as spam and add their cotap domain to your spam service blacklist.

cotap

Cognos – Formatting Output

In creating reports that are usable by people sometimes there is a necessity to change computer given output in order to make it more reader friendly.  In this accounting package there is limitations on the digits available for job numbers and it is up to a 6 digit field with no special characters so a job number might look like 14248. Unfortunately when creating reports most of the people in the organization would expect the format to be 14-248 instead. To achieve this output I used Cast to break apart the values and add hyphen symbol after the second character.

cast(substring([Job#],1,2)+'-'+substring([Job#],3,3),varchar(6))

Basically this takes the 5 digit codes, grabs two characters starting from the first (hence the 1,2) and adds the – symbol followed by grabbing the last there characters starting at item 3 ( 3,3) and then casts this into a varchar that is 6 digits long.

Cognos trials and tribulations – Data Modeling

Having to learn Cognos on the fly has been a challenging yet rewarding experience.  I had done some work with Crystal Reports and Microsoft SSRS in the past and this is a very different beast all together. For one in the environment for my work the Cognos server is hosted by the application provider and Cognos allows a data model to be built, and in this case I do not have direct DB2 SQL access but rather data the way the vendor has modeled it for me. In this case trying to convert from some old reports in a legacy reporting tool for the LOB to a Cognos report proves challenging. Not having direct access to the DB and field names that do not always correlate can make it difficult. Using the ‘Lineage’ tools has helped me to find what I am looking for. To access this, right click on your data object, and choose lineage.

lineage

From here you can choose the ‘Technical View’ tab and see what underlying object in the database this object was derived from.

techview

Java 8 WMI query

In the past using a WMI query to filter computers with Java 7 the query looked like

Select * From win32_Directory where (name="c:\\Program Files\\Java\\jre7" or name="c:\\Program Files (x86)\\Java\\jre7")

which fairly quickly returned results.  Now with Java 8 the folder is in a format of c:\Program Files\Java\jre8.0_xx which causes the query to run for about 10 minutes.  In searching for a better way I came up with

Select * From Win32_Product where name like "Java 8%"

which is reasonably fast, since I wanted both Java 7 and 8 my final query was

Select * From Win32_Product where (name like "Java 7%" or name like "Java 8%")

 

Love and Hate

 

dirrestoremode

If you have been working with Windows servers long enough there are a few things that you both love and hate. Two of these things are likely ‘Directory Services Restore Mode’ and ‘System State Restore’.  When things are looking grim and that Active Directory server will not boot properly or the database is corrupted you are hoping that both of these tools are your best friends and cooperate with your efforts to restore the server. If you can successfully boot into ‘Directory Services Restore Mode’ that is step one and you are feeling excited and nervous at the same time. However make sure you keep your fingers crossed, you pray to whatever deity that you believe in, rub a rabbits foot,  or whatever method of generating luck you have and hope that  you have a valid system state backup that is not too old and that it is not corrupt. Once you locate your backup and successfully restore it and then boot into Windows normally you can than check AD and other functions of the server to make sure they are working properly.  If so you know that you can get a good nights sleep that night. If not you will keep looking for a valid system state backup, image backup or other.  Moral of this story is have good backups and be sure to have system state backups as part of your plan. Even better you will also have image based backups both locally and offsite as a last resort.

Yes I lucked out tonight and a system state restore quickly fixed a down server. Time for bed…….

Unable to log into web interface 3Com 4500 Switch

I had a 3Com 4500 switch I was unable to log into the web interface.  Not being familiar with the cli for this device I was at a bit of a disadvantage. Having done a full factory reset I was still unable to access the web interface.  I had a second unit the same steps were performed on which worked just fine. I noticed the firmware was at a 2006 version so I decided the first step would be to do a firmware update.  I was able to get the required files from the HP site as new as 2012.  I downloaded the files, setup my tftp server.  I use http://tftpd32.jounin.net/ which works really well. And wend forward.  I noticed when I did a dir listing of the device it was missing one of the bootrom files. I updated the files, set the boot options and restarted the device. Still unable to log into web interface, after some google fu I found the procedure to reset web access to the device.

Upgrade process:
first get a list of files:
<4500>dir
Directory of unit1>flash:/
0 -rw- 5195 Feb 04 2007 13:21:21 3comoscfg.def
1 -rw- 642479 Feb 04 2007 13:21:52 s3p02_01.web
2 -rw- 4088266 Feb 04 2007 13:24:15 s3n03_02_00s56.app
3 -rw- 1364 Apr 02 2000 00:47:45 3comoscfg.cfg

Next delete the files an clear recycle bin(there is not enough flash space to store both sets, make sure you have backups)
<4500>delete s3p02_01.web
<4500>delete s3n03_02_00s56.app
<4500>reset recycle-bin
Clear flash:/3comoscfg.cfg ?[Y/N]:y
Clearing files from flash may take a long time. Please wait…
……..
%Cleared file unit1>flash:/~/s3p02_01.web.
Clear flash:/s3n03_02_00s56.app ?[Y/N]:y
Clearing files from flash may take a long time. Please wait…
………………………………………….

Upload new files
<4500>tftp 192.168.18.54 get s3p05_01.web
File will be transferred in binary mode.
Downloading file from remote tftp server, please wait…………………..
TFTP: 1083788 bytes received in 16 second(s).
File downloaded successfully.
<4500>tftp 192.168.18.54 get s3o04_06.btm
File will be transferred in binary mode.
Downloading file from remote tftp server, please wait……….
TFTP: 195022 bytes received in 3 second(s).
File downloaded successfully.
<4500>tftp 192.168.18.54 get s3n03_03_02s168p21.app
File will be transferred in binary mode.
Downloading file from remote tftp server, please wait…………………………………………………………..
TFTP: 4243013 bytes received in 63 second(s).
File downloaded successfully.

Set boot files
<4500>boot boot-loader flash:/s3n03_03_02s168p21.app
The specified file will be booted next time on unit 1!
<4500>boot bootrom flash:/s3o04_06.btm
This will update BootRom file on unit 1. Continue? [Y/N] y
Upgrading BOOTROM, please wait…
Upgrade BOOTROM succeeded!
<4500>boot web-package s3p05_01.web main
<4500>save
The configuration will be written to the device.
Are you sure?[Y/N]y

Reset web interface
[4500]local-user admin
New local user added.
[4500-luser-admin]attribute access-limit 1
[4500-luser-admin]level 3
[4500-luser-admin]service-type telnet level 3
[4500-luser-admin]service-type lan-access
[4500-luser-admin]
<4500>save
The configuration will be written to the device.
Are you sure?[Y/N]y