Code, Cloud, & CTO Insights

Jul 4, 2024

2024 Tech Salary Guides

Here is a list of Australia Tech Salary Guides for 2024.

Mar 19, 2024

Understanding Linear Regression in C#: A Hands-On Implementation

Linear regression is one of the foundational algorithms in statistics and machine learning. It’s used to discover relationships between variables and to make predictions. If you’re exploring machine learning concepts or need a way to analyze trends in your C# applications, understanding linear regression is a great starting point.

What is Linear Regression?

In essence, linear regression attempts to draw the best-fitting straight line through a set of data points. This line represents the underlying trend in the data. Imagine you have data about house prices based on their square footage – linear regression can help you model the relationship between these variables.

Implementing Linear Regression in C#

The basic implementation of an algorithm for linear regression in C# involves calculating the slope and y-intercept of the best-fit line. If you are interested in the math behind it read Least squares regression on maths is fun.

Here’s a simple example:

using System;

namespace LinearRegression
{
    class LinearRegression
    {
        /// <summary>
        /// Calculates the slope of the best-fit line
        /// </summary>
        /// <param name="x">Array of x values</param>
        /// <param name="y">Array of y values</param>
        /// <returns>The slope of the regression line</returns>
        public double CalculateSlope(double[] x, double[] y)
        {
            double xMean = x.Average();
            double yMean = y.Average();

            double numerator = 0;
            double denominator = 0;

            for (int i = 0; i < x.Length; i++)
            {
                numerator += (x[i] - xMean) * (y[i] - yMean);
                denominator += (x[i] - xMean) * (x[i] - xMean);
            }

            return numerator / denominator;
        }

        /// <summary>
        /// Calculates the y-intercept of the best-fit line
        /// </summary>
        /// <param name="x">Array of x values</param>
        /// <param name="y">Array of y values</param>
        /// <param name="slope">The slope of the regression line</param>
        /// <returns>The y-intercept of the regression line</returns>
        public double CalculateIntercept(double[] x, double[] y, double slope)
        {
            double xMean = x.Average();
            double yMean = y.Average();

            return yMean - slope * xMean;
        }

        /// <summary>
        /// Predicts a y value based on a given x value
        /// </summary>
        /// <param name="x">The input x value</param>
        /// <param name="slope">The slope of the regression line</param>
        /// <param name="intercept">The y-intercept of the regression line</param>
        /// <returns>The predicted y value</returns>
        public double Predict(double x, double slope, double intercept)
        {
            return slope * x + intercept;
        }
    }
}
  • The LinearRegression Class: This class encapsulates our algorithm:
  • CalculateSlope: Finds the slope of our best-fit line.
  • CalculateIntercept: Finds where the line crosses the y-axis.
  • Predict: Uses the line’s equation to predict y-values for new x-values.

Using Our Algorithm

using System;
using LinearRegression; 

namespace Example
{
    class Program
    {
        static void Main(string[] args)
        {
            double[] x = { 1, 2, 3, 4, 5 };
            double[] y = { 2, 5, 7, 9, 12 };

            var regression = new LinearRegression();

            double slope = regression.CalculateSlope(x, y);
            double intercept = regression.CalculateIntercept(x, y, slope);

            Console.WriteLine("Equation: y = {0}x + {1}", slope.ToString("F2"), intercept.ToString("F2"));

            double prediction = regression.Predict(6, slope, intercept);
            Console.WriteLine("Prediction for x = 6: y = {0}", prediction.ToString("F2"));
        }
    }
}

In this example, we:

  1. Define sample data (x and y).
  2. Create a LinearRegression object.
  3. Calculate the slope and intercept of the regression line.
  4. Print the line’s equation.
  5. Make a prediction for a new input value.

Checking against R

More to come

Git repo at https://github.com/waynethompson/MlAlgorithms

Mar 18, 2024

The Ultimate Guide to Governance Frameworks

If you are new to leading tech teams, you may have heard about governance but were unsure what it is or how it applies to IT or building software. Put simply, governance gives your team structure, guidelines, and ways to measure success.

From IT governance to risk management, quality control to cybersecurity, there’s a governance framework for nearly every aspect of your organization. This blog post will provide a comprehensive overview of the most widely used frameworks, empowering you to select the ones that best fit your needs.

I’ve compiled this list as a start to understand more about the ecosystem of governance frameworks, and to get a better understanding of where they can best apply to businesses at different stages of their journey.

Core Governance Frameworks

  • COSO (Committee of Sponsoring Organizations): A comprehensive framework for internal control and enterprise risk management (ERM). COSO helps organizations establish sound oversight, effective risk mitigation, and ethical conduct across operations.

  • ISO/IEC 38500: Corporate Governance of Information Technology: Provides principles and a framework for executives and boards to evaluate, direct, and monitor their organization’s use of IT.

  • ISO/IEC 20000: IT Service Management (ITSM): An international standard focused on defining and managing the quality of IT services. It outlines best practices for aligning IT services with business needs.

  • ITIL (Information Technology Infrastructure Library): A widely-used ITSM framework. ITIL provides detailed guidance on processes and practices for delivering and managing IT services throughout their lifecycle.

Project Management Frameworks

  • PRINCE2 (Projects in Controlled Environments): A structured, process-based project management methodology widely used in the UK and Europe. Emphasizes planning, control, and organization.

  • PMBOK (Project Management Body of Knowledge): A guide, rather than a strict methodology, outlining project management knowledge areas and processes. Published by the Project Management Institute (PMI), it’s widely recognized.

Enterprise Architecture Framework

  • TOGAF (The Open Group Architecture Framework): A high-level approach to designing, planning, implementing, and governing an enterprise’s IT architecture. Aids in aligning IT with business goals.

Process Improvement Frameworks

  • CMMI (Capability Maturity Model Integration): Helps organizations improve process maturity across various areas (e.g., development, services). Emphasizes continuous improvement with levels of capability and maturity.

  • NIST (National Institute of Standards and Technology) Cybersecurity Framework: A voluntary framework focused on managing cybersecurity risk. Provides a structure for organizations to assess and improve their ability to prevent, detect, and respond to cyber threats.

  • BiSL/DID (Business Information Services Library/Data, Information, Decisions): Frameworks focused on functional management (BiSL) and information management (DID) in organizations.

Agile Framework

  • Agile: An umbrella term for iterative and incremental software development methodologies (e.g., Scrum, Kanban). Agile emphasizes flexibility, collaboration, and delivering working software frequently.

Quality & Security Frameworks

  • ISO 27000 Series: A family of standards focused on information security management systems (ISMS). ISO 27001 is the core standard outlining ISMS requirements.

  • TQM (Total Quality Management): A management philosophy focusing on continuous improvement across all organizational processes through customer focus.

  • Six Sigma: A data-driven quality improvement methodology aimed at reducing defects and variability in processes.

  • LEAN: Focuses on maximizing customer value while minimizing waste in processes. Emphasizes streamlining and flow.

  • ISO 9000 Series: Quality management standards that provide a framework for consistent product and service quality.

Specialized Frameworks

  • Val IT: A governance framework focused on value creation from IT investments.

  • FAIR (Factor Analysis of Information Risk): A framework for quantifying and managing information security and operational risk.

  • DEFT: A streamlined framework tailored for small to medium-sized enterprises focused on IT governance and processes.

IT Governance Frameworks

  • COBIT (Control Objectives for Information and Related Technologies): Issued by ISACA, COBIT focuses on IT governance and management. It provides goals and processes for aligning IT with business objectives.
  • IT4IT: An open standard from The Open Group, providing a vendor-neutral reference architecture and value chain model for managing the business of IT.

Risk Management Frameworks

  • ISO 31000: Risk Management: Provides guidelines and principles for managing risk across an organization.
  • NIST Risk Management Framework (RMF): A comprehensive framework designed specifically for the US federal government, but with broader applicability, emphasizing cybersecurity risk management.

Security Frameworks

  • PCI DSS (Payment Card Industry Data Security Standard): Mandatory for organizations handling credit card information, ensuring secure processing, storage, and transmission of cardholder data.
  • CIS (Center for Internet Security) Controls: A prioritized set of actions for cyber defense. Provides specific best practices to mitigate common cyber-attacks.

Compliance Frameworks

  • HIPAA (Health Insurance Portability and Accountability Act): US law governing the privacy and security of protected health information (PHI).
  • GDPR (General Data Protection Regulation): European Union regulation focused on data privacy and protection for individuals within the EU.
  • SOX (Sarbanes-Oxley Act): US law centered on financial reporting and internal controls within publicly traded companies.

Industry-Specific Frameworks

  • FFIEC Cybersecurity Assessment Tool: A framework used by US regulatory bodies to assess financial institutions’ cybersecurity maturity and preparedness.
  • NIST Framework for Improving Critical Infrastructure Cybersecurity: Specific for US critical infrastructure sectors, promoting a risk-based approach to cybersecurity.

Conclusion

Governance frameworks are essential for organizations to manage risk, ensure compliance, and align IT with business objectives. They provide a structured approach to managing and improving processes, services, and security. By understanding the various frameworks available, you can select the ones that best fit your organization’s needs and goals.

In coming posts I hope to dive deeper into some of these frameworks, and provide more practical advice on how to implement them in your organization.

Mar 17, 2024

The FAIR Advantage: Data-Driven Decisions for Risk Management.

Imagine a data breach exposing millions of customer records. Traditional risk assessments, often reliant on subjective scales, struggle to communicate the true impact of such an event. Here’s where the Factor Analysis of Information Risk (FAIR) framework steps in. FAIR provides a structured approach to analysing, understanding, and quantifying operational and cyber risk in financial terms (Anderson 2018). FAIR’s data-driven approach to assess and prioritise cybersecurity risks compliments other risk frameworks which propose the necessity to quantify risk without offering guidance on how that should be achieved (FAIR Institute 2024). This formulaic approach to risk simples the process of quantifiable risk management, which is profound as it has been deemed by many as an impossible feat (Freund and Jones 2014).

The FAIR Framework

A challenge of operational risk and information security professionals is the limitations of terminology in the domain of risk related communication (Freund and Jones 2014, ch 3). The intuitive terminology in FAIR simplifies the process of decomposing a risk into an ontological tree diagram of its components. In the process it elicits measurable factors like threat event frequency, vulnerability likelihood, and loss magnitude, which are used to apply probabilistic and loss metrics to a measurement algorithm, resulting in a quantifiable value (Yun, Cho et al. 2015).

FAIR delivers a clear picture of potential financial losses over time. It can calculate not just a single value, but also minimum, most likely, and maximum loss scenarios (Dreyling, Jackson et al. 2021). Additionally, the framework can be further enhanced using advanced data science techniques like Bayesian Networks (Wang, Neil et al. 2020) and Monte Carlo simulations (Hsu, Pan et al. 2023) resulting in accurate predictions.

Impact on Risk Management: FAIR offers several benefits for risk management:

  1. Quantitative Analysis: Moving away from traffic light charts and vague scales, FAIR allows organizations to better understand their exposure by expressing risk in financial terms.
  2. Improved Decision Making: threats can be viewed in terms of financial impact. Allowing stakeholders to understand the potential losses and prioritise based on their potential impact on the organisation. Leading to better resource allocation and more effective risk management.
  3. Consistency and Communication: FAIR provides a framework that reduces uncertainty an improves consistency in risk analysis. It helps risk professionals generate meaningful metrics that can be easily understood and communicated to stakeholders.
  4. Risk prioritisation: By calculating the probable loss associated with each risk scenario, organisations can prioritise risks based on their potential impact. This allows them to focus on their resources on mitigating the most impactful risks first.
  5. International Standard: FAIR creates a standardised model for communicating cybersecurity and operational risk.

Faced with implementing a novel e-service using Amazon Alexa for personal data, the government of Estonia lacked existing risk data (Dreyling, Jackson et al. 2021). They leveraged the FAIR framework to quantify the risk. Consulting cybersecurity experts for breach likelihood, they combined this with global breach data to run simulations. FAIR projected annualized loss exposure ranging from $0 (minimum) to $70,500 (most likely) to $6 million (maximum), providing decision-makers with tangible data to allocate resources for risk mitigation.

In conclusion, by offering a quantifiable and standardized approach to risk communication, FAIR empowers data-driven decision-making and prioritisation for a more secure IT environment.

References

FAIR Institute (2024). “What is FAIR?”. Retrieved 17/03/2024, 2024, from https://www.fairinstitute.org/what-is-fair.

Anderson, B. (2018). FAIR Vulnerability Determined using Attack Graphs. Athens, The Steering Committee of The World Congress in Computer Science, Computer Engineering and Applied Computing (WorldComp): 285-286.

Dreyling, R., et al. (2021). Cyber security risk analysis for a virtual assistant G2C digital service using FAIR model.

Freund, J. and J. Jones (2014). Measuring and managing information risk: a FAIR approach, Butterworth-Heinemann.

Hsu, T.-C., et al. (2023). An Approach for Evaluation of Cloud Outage Risk based on FAIR Model. 2023 International Conference on Engineering Management of Communication and Technology (EMCTECH), IEEE.

Wang, J., et al. (2020). “A Bayesian network approach for cybersecurity risk assessment implementing and extending the FAIR model.” Computers & security 89: 101659.

Yun, J. H., et al. (2015). FAIR-Based Loss Measurement Model for Enterprise Personal Information Breach. Advances in Computer Science and Ubiquitous Computing: CSA & CUTE, Springer.

Jan 10, 2024

Understanding the Essentials of Data Governance in the Digital Age

In today’s fast-paced digital world, data is more than just a collection of numbers and facts. It’s a vital asset that drives decision-making, innovation, and growth in organizations. However, as the volume and complexity of data surge, managing it effectively becomes a daunting challenge. This is where Data Governance steps in, serving as a guiding framework that ensures data is managed efficiently and ethically. In this post, we’ll dive deep into the world of Data Governance, exploring its importance, components, and benefits.

Mar 1, 2023

How to use square brackets '[]' in a string in a SQL query

In SQL, square brackets play a significant role in various scenarios, including escaping table and column names. However, due to their special meaning, SQL Server doesn’t interpret them as usual when they appear within a string. This article aims to shed light on the usage of square brackets in SQL and provide a practical example of escaping them within a string.

Mar 21, 2021

The trade off between Sensitivity and Specificity

When evaluating a model for a binary dependent variable, for instance logistic regression there is always going to be a trade off between Sensitivity and Specificity.

Dec 14, 2020

Creating Azure SQL logins & users that easily replicate to all databases.

Creating Azure SQL logins and database users is fairly simple.

Nov 30, 2020

How to remove x-powered-by header in .net core

Any asp dotnet developer who has had to pass security or OWASP top 10 audits knows that probably the most annoying thing about dotnet is that IIS, and now Azure app service, is the custom headers which give away what server and the .net version you are running on.

Mar 27, 2020

Generating Typescript Enums correctly with NSwag from Swagger docs created with Swashbuckle.

We have been generating typescript classes and data clients for our angular application with NSwag by by pointing it at our AspNetCore api swagger documentation which was generated using swashbuckle.

Mar 21, 2019

Restarting all the Azure App Services in a resource group using PowerShell.

Often I will need to restart the Azure App Services that I use for development and testing. Using the portal web interface can be tedious.

Mar 21, 2019

Debugging a couple of AspNetCore websites and your machine gets slow?

I am developing an asp.net core application comprised of 4 microservices.

Mar 20, 2019

Export git log history to a text file

We had a requirement to export our git log for accounting purposes.

Aug 20, 2018

Unable to configure HTTPS endpoint. No server certificate was specified, and the default developer certificate could not be found.

Today we noticed something strange happen with our dotnet core web apps. They started to throw an error when we ran them on the commandline with dotnet run: