Constants

There are a number of constants that are defined at, or near, the top of the Ivy script (drawdown.ivy) that are intended to be experimented with.

# Rates. All percentages
Igr = 3.5 / 100     # Investment Growth Rate
Sgr = 3.0 / 100     # Savings Growth Rate
Inf = 2.0 / 100     # Annual Inflation Rate
Pcr = 0.25 / 100    # Platform Charge Rate
Tbr = 0.5 / 100     # Tax Band Annual Percentage Increase

# Starting Parameters
N = 30              # Number of years to iterate.
Year1Income = 35000 # Income to draw each year.

Igr and Sgr are the percentage growth rates that are to be assumed will apply over the course of the iterations to investment sources and savings sources. An investment source might be a stocks and shares ISA or a pension fund. A savings source might be a cash savings account or other fixed income instrument, such as a bond. The only real distinction between the two being that you might consider, say, the investment source to have a higher likely annual growth rate than a savings source.

Inf allows a figure to be given for the expected average annual percentage inflation rate over the whole period. Each year the iterations will increase the needed amount of income by this percentage.

Pcr determines the amount of platform charge that is paid each year. Before any withdrawals are made from the sources a platform charge is added to the needed amount of income. The amount of the platform charge is Pcr percent of the remaining asset value at the start of the year. In calculating this asset value we ignore any state pension sources - we assume that they have no platform charge levied on them.

When calculating the tax to be paid each year we consider the tax regimes that are associated with each kind of source. A tax regime, such as capital gains tax or income tax, has a sequence of monetary bands and a percentage tax rate associated with each band. In order to reflect the fact that these bands might increase in value over the years we scale them up by a percentage equal to Tbr at the end of each year.

The constant N determines the number of years over which we will iterate. Our output will contain year end values for each year from 1 to N.

Year1Income is the amount of income that we would like to draw in the first year of the iteration. At the start of each subsequent year this amount will be scaled up by the rate of inflation, Inf. This amount, to which we add any unpaid tax from the previous year and our platform charges, forms the amount of needed income that we will withdraw from the various sources.

Tax Regimes

# Tax Regimes:upper bound and percentage rate
IncReg = (12540 0.0) (50270 20.0) (125140 40.0) (100000000 45.0)
CapReg = (3000 0.0) (100000000 18.0)

A tax regime is implemented as an array of pairs of values. Each pair contains an upper bound amount and a percentage rate. The pairs are assumed to be in order of increasing upper bound amount. Together these pairs form a sequence of tax brackets within which the associated percentage of tax will be charged. For example, the income tax regime above IncReg consists of four bands:

1. £0 to £12,540 at which no tax will be charged.
2. £12,541 to £50,270 at which tax will be charged at the rate of 20%.
3. £50,271 to £125,140 at which tax will be charged at the rate of 40%.
4. £125,141 and above at which tax will be charged at the rate of 45%.

The “and above” bit is represented by the use of a very large upper bound.

Tax Accounts

# Tax Accounts: Tax Regime, taxed amount, tax
IncT = IncReg 0 0
IncT2 = IncReg 0 0
CapT = CapReg 0 0
TA = IncT IncT2 CapT    # Tax Accounts

A tax account is associated with each taxable source. A tax account contains three pieces of information:

1. The tax regime to use when calculating tax on income drawn from sources that use this tax account.
2. The amount, within the current year, on which tax has already been calculated for sources that use this tax account.
3. The amount, within the current year, of tax that has been calculated for sources that use this tax account.

A tax account may be shared between more than one source. For example, a state pension source might share the same tax account as a private pension source so that, when calculating income tax, the income drawn from each source is combined.

The global array TA contains the complete set of tax accounts in use by the program. This is initialised using IncT, IncT2 and CapT as templates.

Sources

Sources are perhaps the central focus of the drawdown program. We model the remaining balance of each of the sources as we make annual withdrawals, and as the assets within each source increase in value each year.

# Sources
S = ("State Pension 1" "State Pension 2" "Savings" "Pension" "GIA") # Source names
B = 0 0 40000 500000 50000                          # Opening balance
TAI = 1 2 0 1 3                                     # Index into Tax Accounts, 0 if non-taxable

# Source flags
IsInv = 0 0 0 1 1                                   # Is an investment source
IsSav = 0 0 1 0 0                                   # Is a savings source
IsSP  = 1 1 0 0 0                                   # Is source a state pension

The global array S contains the names of each of the sources that the program will consider. Several other global arrays (and indeed local variables to operators) have the same shape (length) as S. Ivy does not allow us to model pointers to elements of the source array so, instead, we work with indexes into it and maintain a number of other arrays that keep the same order as the source array.

The global array B contains the opening balance (opening asset value) of each of the sources in S. The balances are arranged in the same order as the sources in S. So, for example, we can see that we start the iteration with a “Savings” balance of £40,000 (the actual currency is not important, but it is assumed to be the same currency for all sources). We do not consider state pensions to have an opening balance. Instead, we populate the “balance” of a state pension source with a suitably scaled value at the start of each year.

The ith entry in the global array TAI gives the index into the tax account array TA at which the tax account for the ith source can be found. It contains a zero for sources that are non-taxable. For example, to find the tax account for the general investment account source (GIA) we note that the GIA source is in index position 5 of the source array S so we look at the contents of index position 5 in the tax account index array TAI and we find the value 3. This tells us that the tax account to use for the GIA source can be found in index position 3 of the tax account array TA.

The lack of pointers means that we have to be careful to ensure that if we change the ordering of sources in S then we also need to make the same ordering changes in other places.

The global constants IsInv, IsSav, and IsSP are set to 1 (true) for those sources that are investment sources, savings sources or state pension sources respectively, otherwise they contain a 0 in the position that corresponds to a source. You can see that a source is marked as belonging to only one of those categories.

State Pension Sources

We handle state pensions slightly differently to the way in which we handle other sources.

# State Pension Sources
SPStartYear = 1 4 0 0 0 # If a state pension, what is its start year?
SPY1 = 10000            # State Pension Year1 Amount
SPAPI = 2.5/100         # State Pension Annual Percentage Increase

We might be modelling the assets of a couple, in which case we might have more than one state pension source and it is quite possible that they do not all come online in the same year. The global array SPStartYear contains the number of the year in which a given state pension source is to come online. It should contain a 0 in the index position for sources that are not state pension sources.

The constant SPY1 contains the annual amount that a state pension will pay in year 1 of the iteration. This amount is scaled up by a percentage governed by the value of the SPAPI constant.

Draw Sequence

# Order by draw sequence
DS = 1 2 3 4 5          # Draw Sequence
S = S[DS]
B = B[DS]
TAI = TAI[DS]
IsInv = IsInv[DS]
IsSav = IsSav[DS]
IsSP = IsSP[DS]
SPStartYear = SPStartYear[DS]

This Ivy implementation of the drawdown program will withdraw amounts from the sources in the order in which they are found in the source array S. Because there are several other global arrays that depend on the ordering of the sources in that array there is a mechanism that allows the sequence to be altered safely. The technique is to change the global draw sequence array DS. This array contains the order of indexes into the source array from which withdrawals will be made. As you can see from the code above the implementation of this is to simply reorder the source array and, at the same time, all of the other global arrays that must be kept in step with it.

If, for example, we wanted to preserve our savings and, instead, draw first from our pension and general investment account we could change the draw sequence to do so:

DS = 1 2 4 5 3          # Draw Sequence

In practice you may not need to make any changes here, but at least it identifies those global arrays whose ordering is dependent on the ordering of the source array.

Withdrawals

The withdrawal functions start with a scalar need amount and an array of remaining balances (one for each of the sources and in source order). The need amount is the annual amount that must be supplied from the sources to satisfy the inflation adjusted income requirements, the platform costs and the previous year’s tax costs in the current year. The job of the withdrawal functions is to step through the remaining balance array withdrawing as much as is available from each source until an amount equal in total to need has been found.

# Withdraw upto need (scalar) from the balances starting at the ith balance.
# Return the new balances
op i withdraw (need balances) =
    balance = balances[i]
    got = 0 max need min balance
    (i == count balances) : got 
    got,  (i + 1) withdraw ((need - got) balances)

# Withdraw need (scalar) from the balances returning the new balances
op withdraw (need balances) =
    b = 1 withdraw (need balances)
    b

The dyadic version of withdraw is passed (an array containing) a scalar need amount and an array of remaining balances in source order as its right hand argument and a starting index into the balance array as its left argument. The balance array is guaranteed to have the same length as the source array. The balance in the ith index of the array is the remaining balance of the source in the ith position of the source array S.

The aim of withdraw is to find an amount equal to need by withdrawing as much as possible from each of the source balances in order. It returns a withdrawal array of length equal to (count S) - (i-1) with the amount in the first position being the amount that is to be withdrawn from the ith source. Here count S is the length of the source array.

The function focuses on the ith balance. The value got is calculated by first taking the minimum of what we need and the available balance. The restriction that got must be positive is there to protect against being called with a negative balance or need.

If i is equal to the length of balances it returns the scalar got. Otherwise it returns the result of catenating the value of got with the result of calling withdraw with an index of (i+1) and a need amount that has been reduced by the amount that we have got from the source in the ith position.

The monadic version of withdraw returns the result of calling this function with an index of 1.

Tax

Given the withdrawal array (which is the amount that we propose to withdraw from each source) the job of the tax operators is to calculate an array of tax amounts that will be raised as a result of withdrawing the amounts w from the sources.

The amount of tax due from a particular source will depend upon the withdrawn amount, the tax regime associated with the tax account that is associated with the source, and also the amount that has previously been withdrawn from other sources using the same tax account. To be able to calculate this latter figure the tax account contains a running total of the amount withdrawn from all of the sources associated with the tax account. This running total is reset at the end of each year.

op second a = 1 take 1 drop a

# Calculate the amount of tax due for the amount 'a'
op tr taxDue a =
    u = first@ tr                           # upper bounds
    r = second@ tr                          # tax rates
    ba = (a min u) - (a min -1 drop (0,u))  # amounts in each bracket
    t = +/(ba * r/100)                      # sum of (bracket amount times rate)
    t

# Calculate tax on the ith source given w[i] withdrawn.
# Return zero for non-taxable sources.
# The tax account contains the already withdrawn amount from other sources in the same tax account.
# Update the tax account with the amount withdrawn and tax calculated.
op i taxOn ws = 
    tai = TAI[i]                        # tax account index
    (tai == 0): 0
    ta = TA[tai]                        # tax account
    tr = ta[1]                          # tax regime
    w0 = ta[2]                          # amount that tax has already been calculated on (from other sources)
    w = ws[i]                           # withdrawn amount from this source
    t0 = ta[3]                          # tax already calculated (from other sources)
    t = (tr taxDue (w0+w)) - (tr taxDue (w0))   # additional tax
    TA[tai] = ta[1] (w + w0) (t + t0)           # accumulate taxed amount and tax (raised)
    t

# Calculate the tax to pay on each element of w
op taxOn w =
    (iota count S) @taxOn w                 # tax due

The monadic taxOn operator is called with an array of withdrawals (one for each source). It calls the dyadic taxOn operator with a source index as its left argument.

The dyadic taxOn operator focuses on the ith source. It determines the tax account index by indexing into the tax account index array TAI and so finds the tax account.

From the tax account it extracts the tax regime tr, the amount of withdrawals that we have already calculated tax on using this tax account w0, and the tax that has already been calculated as being due on this tax account from other sources t0.

Then, using the amount that we have withdrawn from this ith source w it calculates the difference between the tax that would be due on a withdrawal of (w0+w) and an amount that would be due on a withdrawal of just w0 - this is the additional tax t.

Finally it updates the tax account with the new values for the amount that has been taxed, w+w0, and the amount of tax raised, t+t0.

The taxDue operator takes a tax regime as its left argument and a scalar amount as its right argument. It returns the scalar amount of tax due, t, as a result of withdrawing the amount a.

It begins by extracting the array of upper bounds, u, and the array of associated tax rates, r from the tax regime.

It calculates the size of each of the tax brackets by subtracting u shifted right by one position (and with a zero prepended) from itself. But by carefully limiting the maximum value of either term to be a it results in an array of values of the same length as u but which contains the amount of a that falls into each bracket, ba.

It is then easy to calculate a scalar tax value by summing over the product of the rate in r and the amount in ba. This value, t, is then returned.

Year Start and Year End

At the start of the year we initialise the state pension balances and scale up the investment and savings balances using the appropriate scaling constants.

# Start year n with balances b.
# Return new balances (after any scaling up)
op startYear (n b) =
    ids = iota count b
    # set state pensions
    i = (IsSP and (n >= SPStartYear))           # identify state pensions
    b[i sel ids] = SPY1 * (1 + SPAPI) ** n - 1  # set their scaled up balance
    #
    # scale up investments and savings (apart from in year 1)
    (n == 1): b
    b[IsInv sel ids] = b[IsInv sel ids] * 1 + Igr 
    b[IsSav sel ids] = b[IsSav sel ids] * 1 + Sgr 
    b

# Reset the counters in the tax account with index 'tai'
op resetAndScaleTaxAccount tai =
    ta = TA[tai]                # tax account
    tr = ta[1]                  # tax regime
    m=mix tr                    # as an nx2 matrix. first column contains the bands.
    m[;1] = m[;1] * 1 + Tbr     # scale the bands
    tr = split m                # return to vector shape
    TA[tai] = tr 0 0            # set scaled tax regime, and zero taxed amount and tax counters

op endYear (n b) =
    resetAndScaleTaxAccount@ iota count TA
    b

State pension balances are assumed to be fully withdrawn each year.

At the end of the year we scale the bands in the tax account tax regime up using the appropriate percentage constant and we reset the taxed amount and tax raised counters in the tax account

Yearly Iteration

UnpaidTax = 0
# Iterate year n with starting balance b. Return balance, withdrawals, and tax incurred at year end.
op n year b =
    need = Year1Income * (1 + Inf) ** n - 1     # scale by inflation.
    need = need + UnpaidTax                     # add in unpaid tax from the previous year.
    UnpaidTax = 0
    b = startYear n b   
    pc = (+/ b[(not IsSP) sel iota count b]) * Pcr  # platform charge
    need = need + pc            # add in platform charge
    w = withdraw (need b)       # withdrawn amounts that sum to need
    t = taxOn w                 # calculate tax according to the tax regimes
    UnpaidTax = +/ t            # set UnpaidTax for payment next year.
    b = endYear n b
    (b - w)  w t                # end of year balance, withdrawals, and tax
    

# Return the balance after maxN years
# given the starting balances, withdrawals and taxes
# at the end of the previous 1...n years (n <= maxN).
op maxN years (n bs ws ts) =
    b w t = n year bs[n]        # balances, withdrawals and tax at the end of year n
    bs = bs ,% b                # append as row to bs
    ws = ws ,% w                # append as row to ws
    ts = ts ,% t                # append as row to ts
    (n == maxN): bs ws ts       # on reaching year maxN, return the accumulated rows.
    maxN years (n+1) bs ws ts   # otherwise recurse to return years (n+1) and above.

The work of calculating the effect of a single year is performed by the year operator. Given a starting balance array b (of length equal to the number of sources) and a year number n we calculate the end of year balance array, and return it with the withdrawal array and tax array.

The amount of income needed for the year is calculated by scaling up the Year1Income constant by the rate of inflation compounded over (n-1) additional years. To this we add any (unpaid) tax that was raised by withdrawals in the previous year, and the platform charges.

Unpaid tax is maintained in a global variable, a rather clumsy mechanism that requires that the year operator is called with consecutive values of n - which it is. The global UnpaidTax is initialised immediately above the year operator to stress its local nature.

The platform charge is determined as a percentage of the balance of the non-state pension sources.

The years operator recursively iterates through years from n up to maxN years (where n < maxN) calling year for each new value of n and appending the returned balance, withdrawal and tax arrays, b, w, and t, as new rows on the bs, ws, and ts matrices.

The end result is that the data for year n appears in the nth row of each of these matrices. Once we reach year n we return the completed matrices and no further recursion occurs.

Execution and the Output of CSV Data

# Unpack the year, source name, amount withdrawn, tax raised,
# and year-end balance for each combination of year and source. 
op (n i) byYearSource (s bs ws ts) = 
     n s[i] ws[n;i] ts[n;i] bs[n;i]

op a csv b = (text a) , ',' , (text b)


# Run for n years with b as the opening balance.
op run (n b) = 
    bs = 1 (count b) rho b  # matrix with b as the only row
    ws = 0 (count b) rho 0  # empty matrix of the same shape 
    ts = 0 (count b) rho 0  # empty matrix  of the same shape
    bs ws ts = n years 1 bs ws ts
    bs = 1 drop bs          # drop the opening balance
    year_source = , (iota n) o., (iota count S)
    M = year_source @byYearSource S bs ws ts
    print "Year, Source, Withdrawn, Tax Raised, Balance"
    mix csv/mix M

run N B

The monadic operator run takes a single array parameter which contains two values: a number of years, n, for which to run the iteration, and an array, b, which contains the opening balance for each of the sources.

The aim of run is to use years to iterate over the given number of years using the starting balances. It does this by calling the dyadic operator years with a starting year of 1 and an ending year of n and capturing the returned matrices. The returned matrices contain the end of year balances, within year withdrawal amount, and within year tax amount. Year n data is in the nth row of each matrix.

There is a little trick in the bootstrapping of the balances array: the years operator expects to find the opening balances for year n in the nth row of the bs matrix that is passed to it as an argument. So we initialise bs with the passed in opening balances and then when years returns we drop the initial opening balance from bs so that it contains the closing balances for year n in the nth position and has a length of n - the same as the lengths of both ws and ts.

To output the CSV we use the monadic print operator to output our textual header. We construct a single array year_source that contains an entry for each combination of year (1 to n) and source index (1 to count S). We do this with the outer join operator o. applied to the catenation operator (dyadic ,) and then ravel the whole thing into a one-dimensional array with the ravel operator (monadic ,).

year_source = , (iota n) o., (iota count S)

This gives us a vector that, for n = 30 and 4 sources, looks like this:

(1 1) (1 2) (1 3) (1 4) (2 1) (2 2) (2 3) (2 4) (3 1)...(30 4)

We construct a dyadic operator byYearSource that takes one of these year-and-source tuples as its left argument and the source, balance, withdrawal, and tax matrices as its right argument and which returns a vector containing the values of the CSV data that we would like to see on each line Such a line would correspond to year n and source (index) i.

op (n i) byYearSource (s bs ws ts) = 
     n s[i] ws[n;i] ts[n;i] bs[n;i]

We invoke “each left” on this operator to generate a matrix containing the CSV data that we would like to see in our output.

M = mix year_source @byYearSource S bs ws ts

Our matrix M contains a row for each year and a column for each of the 5 data values returned by byYearSource - year, source name, withdrawal, tax and balance.

We reduce across the rows with our csv operator which gives us a vector with an element for each value of year_source and with the data values separated by a comma. We lay these out one per line by applying mix again.

mix csv/M

Where the csv operator is defined (above) as:

op a csv b = (text a) , ',' , (text b)

Finally we invoke the run operator passing it, as values of n and b, the global constants N and B which were set near the top of the Ivy script.

The resultant text is printed to standard output and we capture it manually into a file, perhaps by invoking ivy from the shell, like this:

ivy drawdown.ivy >drawdown.csv

The output looks something like this:

Year, Source, Withdrawn, Tax Raised, Balance
1,State Pension 1,10000,0,0    
1,State Pension 2,0,0,0        
1,Pension,26475,4787,473525    
1,GIA,0,0,50000                
1,Savings,0,0,40000            
2,State Pension 1,10250,0,0    
2,State Pension 2,0,0,0        
2,Pension,31695,5868,458404    
...

Such data can easily be read into a tool for analysis. Different outcomes can be modelled by changing the values of various constants and re-running the iteration:

• N, to give the number of years on which to iterate.
• B, to modify the opening balances.
• Igr, Sgr, Inf, Pcr, Tbr, to see the effect of different rates.
• The upper bounds and percentage tax rates of the different kinds of tax regime.
• The order in which income is withdrawn from the sources in S (using the draw sequence in DS).
• The state pension starting years and annual amounts and increases in SPStartYear, SPY1 and SPAPI.

More complicated scenarios can be modelled with the Go version of Drawdown.

Example Data

The examples shown in this document are based on a simple drawdown strategy that simulates the impact of a withdrawal amount of £35,000 taken annually over a 30 year period for a couple beginning their pension drawdown and who:

• Each have a state pension of £10,000 per year with State Pension 1 starting immediately (year 0) and State Pension 2 starting after one year (year 1). The state pension is assumed to increase at the rate of 2.5% per year.
• Have £40,000 of savings.
• Have £40,000 in an ISA.
• Have private pensions: “Pension 1” of £350,000 and “Pension 2” of £150,000.
• Have £50,000 in a general investment account (GIA).

In this model, private pensions are assumed to come with a tax-free lump sum of 25% as soon as they enter drawdown - they are effectively split into a taxable drawdown pension of 75% of the original value and a lump sum of 25% of the original value.

When drawdown is run in normal mode (i.e. without the “-s” flag) it uses a fixed set of rate parameters to calculate the situation at the end of each of the 30 years. When run with the “-s” flag it runs the same iterations to completion for every combination of values of the rate parameters.

When run in normal mode the following parameters are used:

const (
    InvestmentGrowthRate     = 3.5  // %
    SavingsGrowthRate        = 3.5  // %
    AnnualInflationRate      = 2.5  // %
    PlatformChargeRate       = 0.25 // The % charge for using a platform as a percentage of the balance.
    TaxBandAnnualPctIncrease = 0.5  // %
)

When run in summary mode the calculation is performed over every combination of the following values (taken at suitable steps):

• Investment Growth Rate from 0% to 8%.
• Savings Growth Rate from 0% to 8%.
• Annual Inflation Rate from 2% to 5%.
• Platform Charge Rate from 0.1% to 0.5%.
• Tax Band Annual Percentage Increase from 0% to 2%.

These values are built into the program, as are the strategies themselves, but after making a change, a simple recompilation can be performed in less than a second. In this initial version of the program both the parameters and the strategies are altered by editing the Go code. A later version might move the rate parameters to a textual format file and allow the choice from a number of fixed strategies.

Examples of Output - Normal Mode

Actual output of the program takes the form of a CSV file. Different formats are generated for the normal and summary mode of operations but both are suitable for use in another program, such as Microsoft Excel, for the purposes of analysis and visualisation.

Figure 1 displays an Excel stacked bar-chart that shows, for each of the 30 years for which the iterations were run, the total balance of funds remaining broken down by each source. You can see that the tax-free lump sums (TFLS) shown in orange and blue are exhausted by years 6 and 8 respectively. Also noticeable, is that the ISA balance grows in size up until about year 7, after which is decreases each year until it is used by by year 14. After year 14 the balance is contained wholly within the two pension funds (and also by a couple of state pensions that are not shown in this chart because we don’t consider them to have a balance).

Figure 2 shows the income drawn from the sources each year as a stacked bar-chart with colours being used to identify the size of the contribution from each source. You can see that from year 16 onwards all of the income is coming from the combination of the two state pensions and pension 1. We can also see how a contribution from State Pension 2 didn’t occur until year 1 (because our strategy contains that information) and that because of that a larger amount was drawn from the ISA in year 0.

If you are wondering why the ISA, given that it is used to provide income through to year 15, continues to increase in size, it is because our strategy contains an action that transfers the maximum amount allowed each year from the savings account and the tax free lump sums into the ISA so that it can earn interest over the course of the year in a tax efficient way.

Figure 3 shows a different view of the same information. This time in Pivot Table format you can see, for each of the 30 years, the amount drawn from each source to arrive at the desired annual total figure. The annual total required is calculated from an inflation adjusted annual amount which in this example was set at £35,000 per year. The reason why the total at the end of year 0 is greater than our chosen initial amount is that it is increased to include platform costs for hosting the drawdown pensions and investment and savings accounts. It will also, in later years, be increased by any amount of tax that is incurred when drawing sums from the non-tax-free instruments.

Figure 4 shows that very little tax needs to be paid for the first 15 years because we are mostly drawing income from tax-free sources or, as in the case of the GIA (general investment account), we are keeping our annual withdrawals below the appropriate capital gains tax allowance

Normal Mode Output - Drawdown.csv

The examples above are all built using the normal mode output found in drawdown.csv which looks like this (years 2 to 28 are omitted):

Year,Source,Amount,Tax,Tax Raised,Balance
0,"State Pension 1",10000,0,0,0
0,"State Pension 2",0,0,0,0
0,"Pension 1",0,0,0,262500
0,"TFLS 1",1000,0,0,86500
0,"Pension 2",0,0,0,112500
0,"TFLS 2",0,0,0,37500
0,"Savings",0,0,0,20000
0,"ISA",22600,0,0,37400
0,"GIA",3000,0,0,47000
1,"State Pension 1",10250,0,0,0
1,"State Pension 2",10250,0,0,0
1,"Pension 1",0,0,0,271687
1,"TFLS 1",1025,0,0,88502
1,"Pension 2",0,0,0,116437
1,"TFLS 2",0,0,0,38812
1,"Savings",0,0,0,200
1,"ISA",12898,10,0,46311
1,"GIA",3074,0,10,45570
...
29,"State Pension 1",20464,0,4092,0
29,"State Pension 2",20464,0,1197,0
29,"Pension 1",43854,9304,4015,73198
29,"TFLS 1",0,0,0,0
29,"Pension 2",0,0,0,305066
29,"TFLS 2",0,0,0,0
29,"Savings",0,0,0,0
29,"ISA",0,0,0,0
29,"GIA",0,0,0,0

Each row gives the end-of-year state for a given year and source:

• The Amount column is the amount of money that was drawn from that source (including any tax) in the given year.
• The Tax column is the amount of tax that was paid from this source in the given year.
• The Tax Raised column is the amount of tax incurred as a result of drawing from this source in the given year.
• The Balance column is the amount remaining in the source at the end of the given year.

State pensions are assumed to have infinite balance but we return a value of zero for want of a better representation.

You can see from the year 29 rows above that at the end of the 30 year period we are left with a non-zero balance in both of the private pensions as well as still having the two state pensions. We can infer from this that this particular drawdown strategy has preserved the pension pots at the expense of the savings and ISA funds. An alternative strategy might be to drawdown from the pensions first, leaving the tax-free sources until later. This tool allows us to compare the outcome from such different strategies.

Summary Mode Output - Summary.csv

When we run the drawdown program with the “-s” flag we generate a summary output CSV file. In the summary output each row represents a complete iteration of the strategy over the given period of years (in this example 30 years). For each combination of the input rates we obtain values for each of the four output measures of interest.

Investment Growth Rate,Savings Growth Rate,Annual Inflation Rate,Platform Charge Rate,Tax Band Annual Percentage Increase,Total Withdrawn,Tax Paid,Final Balance,Final Year
igr_0.00, sgr_0.00, air_2.00, pcr_0.10, tbi_0.00, 1457098, 81846, 0, 28
igr_0.00, sgr_0.00, air_2.00, pcr_0.10, tbi_0.50, 1498026, 73899, 0, 29
igr_0.00, sgr_0.00, air_2.00, pcr_0.10, tbi_1.00, 1498026, 69204, 0, 29
igr_0.00, sgr_0.00, air_2.00, pcr_0.10, tbi_2.00, 1498026, 56253, 0, 29
igr_0.00, sgr_0.00, air_2.00, pcr_0.25, tbi_0.00, 1457098, 78323, 0, 28
igr_0.00, sgr_0.00, air_2.00, pcr_0.25, tbi_0.50, 1457098, 76125, 0, 28
igr_0.00, sgr_0.00, air_2.00, pcr_0.25, tbi_1.00, 1457098, 66809, 0, 28
igr_0.00, sgr_0.00, air_2.00, pcr_0.25, tbi_2.00, 1498026, 55659, 0, 29
...

The example above shows the first few rows from our summary output file. Each row represents the end result of an iteration over 30 years and gives the following final values:

• The Total Withdrawn column is the sum of the amount withdrawn from all of the sources over the iteration (including tax).
• The Tax Paid column is the sum of the amount of tax paid from all of the sources over the iteration.
• The Final Balance column is the sum of the balance remaining across all of the sources at the end of the iteration.
• The Final Year column is the year in which the sources were unable to supply the required needs, or the intended final year of the iteration (if a non-zero balance remains).

These values are given for an iteration for each of several values of input rates:

• The Investment Growth Rate column contains the percentage rate at which non-savings investments are assumed to grow each year.
• The Savings Growth Rate column contains the percentage rate at which savings investments are assumed to grow each year.
• The Annual Inflation Rate column contains the rate at which the needed annual income is increased each year.
• The Platform Charge Rate column contains the percentage of the total balance of a source that is charged in fees by the platform each year.
• The Tax Band Annual Percentage Increase column contains the percentage rate at which the tax band rates are assumed to be increased each year.

The summary mode output file summary.csv can be easily used by analysis programs, such as Microsoft Excel, to present this data in a useful format.

Examples of Output - Summary Mode

We can import the summary mode data into Excel and visualise it using a combination of pivot tables and charts.

Figure 5 shows the effect on the final balance of different values of platform charge rate and annual inflation rate. By choosing to display the average final balance figure we can use slicers to select one or more values of each of the other rates and we obtain the average final balance over them. In this example we have averaged over investment and savings growth rates of 3% and 4%, and have fixed the tax band annual percentage increase at 0.5%. We can see from both the pivot table and the chart that with this strategy, its starting balances, annual withdrawal amount and our selected growth rates, the final balance is very dependent on the annual inflation rate. In particular, an annual inflation rate over the retirement period of 4% or higher will mean that we run out of funds before the 30 year period ends.

Another observation is that the level of charges levied by the platform also has a big impact on the final balance. Typically, unless we run out of funds, this is of the order of £150,000.

We can take a closer look at how long our sources will last under different input rates.

Figure 6 shows that an annual inflation rate of 4% will result in our funds lasting somewhere between 24 and 26 years depending upon the platform charge rate (all other rates being as defined by the sliders).

There are a number of views of the data that can be obtained and which are all useful in understanding how a particular drawdown strategy will unfold given its input rates and starting balances.

The Drawdown Strategy

The elements that define a drawdown strategy are held in a DrawScenario struct.

type DrawScenario struct {
    Sources                  []*Source
    DrawSequence             []*Source
    TaxPaymentSequence       []*Source
    TaxAccounts              map[*Source]*TaxAccount
    TaxRegimes               []*TaxRegime
    Actions                  []func(year int, need int64, s *DrawScenario)
    InflationLinkedVariables []*int64
    Rates                    DrawRates
}

// Source represents something from which income can be drawn.
// This might be a savings account, an investment account, or a pension, for example.
type Source struct {
    Name           string
    balance        int64                             // The amount of money currently in the source.
    year           int                               // The current year (origin zero) - decisions might be based on this.
    startYear      func(year int)                    // Called at the beginning of each year typically to set the opening balance (year origin is zero).
    endYear        func(year int)                    // Called at the end of each year.
    makeWithdrawal func(amount int64) []SourceAmount // nil, else it returns the amount withdrawn from the source.
}

// Type SourceAmount represents an amount of money associated with a given source.
type SourceAmount struct {
    Source *Source
    Amount int64
}

drawSequence := []*drawdown.Source{
        is_state_pension_1,
        is_state_pension_2,
        drawdown.Seq(&capitalGainsTaxAllowance, is_gia),
        drawdown.Seq(&v1000, is_pension_1_tfls, is_savings, is_pension_2_tfls),
        is_isa,
        is_pension_1_tfls,
        is_pension_2_tfls,
        is_pension_1,
        is_pension_2,
        is_savings,
        is_gia,
    }

// A TaxAccount records the amount of a money on which tax has already been calculated.
// and also the calculated total amount of tax due on that amount.
type TaxAccount struct {
    Name        string
    regime      TaxRegime
    taxedamount int64 // The amount of money on which tax has already been calculated.
    tax         int64 // The total amount of tax that has been calculated to be due on that amount.
}

taxAccounts := map[*drawdown.Source]*drawdown.TaxAccount{
    is_state_pension_1: incomeTaxAccount1,
    is_state_pension_2: incomeTaxAccount2,
    is_pension_1:       incomeTaxAccount1,
    is_pension_2:       incomeTaxAccount2,
    is_gia:             capitalGainsTaxAccount,
}

// TaxRegime describes the rates of tax that are charged on increasing amounts.
// Commonly the first RateBound in the slice might represents a tax-free allowance (i.e. its rate is zero).
type TaxRegime struct {
    Rates []RateBound
}

// RateBound contains a rate and an upper bound on the amount for which the rate applies.
// A slice of RateBound is used to describe a tax regime.
// In such a slice, subsequent upper values must be strictly increasing.
// Rates are represented as a decimal percentage. For example, 10.1 for 10.1%.
type RateBound struct {
    upper int64
    rate  float64
}

incomeTaxRegime := drawdown.NewTaxRegime([]drawdown.RateBound{
        drawdown.NewRateBound(12540, 0.0),
        drawdown.NewRateBound(50270, 20.0),
        drawdown.NewRateBound(125140, 40.0),
        drawdown.NewRateBound(drawdown.HighUpperBound, 45.0),
    })

actions := []func(year int, need int64, s *drawdown.DrawScenario){
    func(year int, need int64, s *drawdown.DrawScenario) {
        //fmt.Println("year", year, "need", need)
    },
    func(year int, need int64, s *drawdown.DrawScenario) {
        // Every year make the maximum annual ISA contribution from the savings accounts.
        drawdown.Transfer(&annualMaximumIsaContribution, is_isa, is_savings, is_pension_1_tfls, is_pension_2_tfls)
    },
}

// Inflation linked variables
incomeTaxAllowance := incomeTaxRegime.TaxFreeAllowance()
capitalGainsTaxAllowance := capitalGainsTaxRegime.TaxFreeAllowance()
annualMaximumIsaContribution := int64(20000)
v1000 := int64(1000)

allInflationLinkedVariables := []*int64{
    &annualMaximumIsaContribution,
    &incomeTaxAllowance,
    &capitalGainsTaxAllowance,
    &v1000,
}

type DrawRates struct {
    InvestmentGrowthRate     float64
    SavingsGrowthRate        float64
    AnnualInflationRate      float64
    PlatformChargeRate       float64
    TaxBandAnnualPctIncrease float64
}

    is_savings := drawdown.NewSavingsAccount("Savings", SavingsInitialBalance, &s.Rates.SavingsGrowthRate)
    is_isa := drawdown.NewInvestmentAccount("ISA", IsaInitialBalance, &s.Rates.InvestmentGrowthRate)

    is_pension_1 := drawdown.NewInvestmentAccount("Pension 1", 0.75*Pension1InitialBalance, &s.Rates.InvestmentGrowthRate)
    is_pension_1_tfls := drawdown.NewSavingsAccount("TFLS 1", 0.25*Pension1InitialBalance, &s.Rates.SavingsGrowthRate)

Summary

There are some obvious limitations to the drawdown model that could be improved upon:

1. It is almost certainly unreasonable to be able to predict any of the rates with a degree of accuracy over a 30 or 40 year period - especially the rate of inflation.
2. It would be nice to move the configuration of the drawdown strategy out of code and into a textual format (such as JSON) so that the tool could more readily compare multiple strategies as well as multiple rates.

A partial solution to the first limitation is to use the summary mode to examine the effect of a range of values for a given rate. A mitigating factor for the second limitation is that, assuming you are a Go programmer, it is easy and quick to make a change to the strategy by editing the code.

Nevertheless, the tool can be used to get a feel for how comfortably (or not) your desired annual withdrawal can be satisfied from the set of investments and savings that you have at retirement. And, by making changes to the order of withdrawal, by altering the draw sequence, you can experiment with different strategies.

Source Code

The Go code for Drawdown can be found here: https://github.com/Vextasy/Drawdown_Go.

The strategy used in the examples in this document can be found in the source file scenario/simple.go.

Techniques

In addition to using the very nice interface that go-anthropic provides to allow interaction with Claude using Go, this application makes use of three techniques that are described in the Claude documentation here:

1. Embedding
2. Tool Use
3. XML Tags

Embedding

The Claude documentation describes embeddings as “numerical representations of text that enable measuring semantic similarity”. So, strictly, in this application we don’t do proper embedding. But I was keen to see how embeddings could be used to incorporate data from a local database into the interaction with an AI and the technique that is used here is similar to what would be done when performing proper embedding.

An embedding model can be trained on a local dataset and will return documents that most closely match a given query. These documents can then be passed to Claude as part of the prompt generation process in order that Claude might use the information that they contain to help answer the original query.

Claude doesn’t have its own embedding model but, rather, works with an external embedding mechanism. Currently they recommend using Voyage AI.

For this simple application, our local database is a collection of TV programme schedules. Each programme has a name, a channel, a date, a time and a description. We encourage Claude to ask for details of TV programmes that are scheduled to be broadcast between a given range of dates. We could equally well have asked Claude to request programmes based on a more complex query, such as by genre or subject matter but then we would need to consider using a proper embedding database in order to provide a good response. So by keeping the local database queries to just a range of dates we can handle selection from the database with a very basic model.

Tool Use

Claude allows user-defined tools to be described as part of a request. A tool can be thought of as a function that Claude might call to request certain information. As well as giving the tool a name, and a description that describes to Claude what the tool can be used for and how to use it, it also enumerates any parameters that Claude should supply when calling the tool.

If Claude thinks that it could benefit from using a particular tool it will return a response to its initial query containing a request to call the tool and will supply the necessary tool parameters. A request to make use of a tool will appear in a response with a content.Type == anthropic.MessagesContentTypeToolUse. And the parameters can be obtained from the content.MessagecontentToolUse field.

In an application making real use of an embedding database we might ask Claude to supply a query that could be made to the database to return a set of documents with which to enhance the subsequent request to Claude. In our rather simple example the tool use request will contain the date range between which Claude would like to know about programmes from our TV database.

The tools available to Claude are described in the Tools section of the initial message request. Our get_tv_programmes tool is described like this:

Tools: []anthropic.ToolDefinition{
    {
        Name: "get_tv_programmes",
        Description: `Gets information about the TV programmes that will be aired between the given from_date and to_date inclusive.
            If to_date is not provided, it will default to the same date as the from_date.
            It should be used when the user asks about TV programmes and we want to know which programs will be shown in the near future including today.
            Programme information will be returned in XML tags with the following format:
                <Programmes>
                    <Programme>
                        <Channel>Channel Name</Channel>
                        <Name>Programme Name</Name>
                        <Date>YYYY-MM-DD</Date>
                        <Time>HH:MM</Time>
                        <Description>Programme Description</Description>
                    </Programme>
                </Programmes>
            The outer level <Programmes></Programmes> tag will be omitted if the result is empty,
            but will otherwise contain one or more <Programme></Programme> tags.
            The <Location></Location> tag contains the TV channel on which the program will be shown.
            `,
        InputSchema: jsonschema.Definition{
            Type: jsonschema.Object,
            Properties: map[string]jsonschema.Definition{
                "from_date": {
                    Type:        jsonschema.String,
                    Description: "The start date of the TV programmes search. Format: YYYY-MM-DD",
                },
                "to_date": {
                    Type:        jsonschema.String,
                    Description: "The end date of the TV programmes search. Format: YYYY-MM-DD",
                },
            },
            Required: []string{"from_date"},
        },
    }},

It makes it quite clear which parameters will be required and the format in which information will be returned from the tool.

XML Tags

Anthropic suggest that good use is made of XML tags to supply data to Claude or to return data from Claude. This is achieved by being explicit about the format of such input or output data in the prompt that is constructed to send to Claude.

In our case we construct the following initial prompt:

You are given information about TV programmes enclosed between <Programmes></Programmes> tags.
The date today is ` + time.Now().Format("2006-01-02") + `, a ` + time.Now().Weekday().String() + `.
You are an expert at suggesting TV programmes.
The user would like to have some suggestions about what TV programmes to watch but has the following strict request:<Request>` + desire + `</Request>.
Your task is to provide a list of programme suggestions in XML format using the following tag structure for each selection:
<Suggestion><Channel>Channel Name</Channel><Name>Programme Name</Name <Date>YYYY-MM-DD</Date><Time>HH:MM</Time><Description>Programme Description</Description></Suggestion>
Wrap the list of <Suggestion></Suggestion> tags in a <Suggestions></Suggestions> tag.
Do not output any other information either before or after the <Suggestions></Suggestions> tags.
If the user does not request that programs begin within a particular date range then use a range that begins today and ends 1 week later.
Return at most 10 suggestions.
If you have no suggestions then return the <Suggestions></Suggestions> tags with no content.
Before answering re-check that all of the user's requests have been met.
If you are not sure that a particular suggestion satisfies the user's Requests then do not return that suggestion.
If the user has requested programmes on a particular day of the week or days of the week check that only programmes on those days are returned.
`

The prompt explains to Claude that it can find programme schedule information between <Programmes> and </Programmes> tags and that it should return information between <Suggestions> and </Suggestions> tags. This makes it easy for Claude to read the results of its use of a tool and for us to read the output of the final interaction with Claude.

Constructing the TV Database

In order to construct the database of TV schedules we made use of Claude to generate some fictitious TV programmes and to assign them a schedule. The prompt needed to perform that task was:

I would like you to generate some fictitious TV listings data for me.

The data will be in XML format using the following tags:

<Programmes>
    <Programme>
        <Channel>Channel Name</Channel>
        <Name>Programme Name</Name>
        <Date>YYYY-MM-DD</Date>
        <Time>HH:MM</Time>
        <Description>Programme Description</Description>
    </Programme>
<Programmes>

Each programme is described by the contents of a pair of <Programme></Programme> tags.

Valid values for Channel are: BBC1, BBC2, BBC3, BBC4, ITV, Channel 4.

Here are some examples:

<Programmes>
    <Programme>
        <Channel>BBC1</Channel>
        <Name>The Night of the Hunter</Name>
        <Date>2024-06-20</Date>
        <Time>18:00</Time>
        <Description>A film about the life of a hunter and his family who travel from location to location in search of food.</Description>
    </Programme>
    <Programme>
        <Channel>BBC2</Channel>
        <Name>Would I Lie to You?</Name>
        <Date>2024-06-20</Date>
        <Time>19:00</Time>
        <Description>A comedy quiz in which contestants have to determine if the others are lying or telling the truth.</Description>
    </Programme>
    <Programme>
        <Channel>BBC4</Channel>
        <Name>Easter Island Origins</Name>
        <Date>2024-06-20</Date>
        <Time>19:00</Time>
        <Description>Documentary exploring how new evidence is challenging everything we thought we knew about Easter Island and the nearly 900 giant stone statues scattered across this remote Pacific island. They are some of the most famous and mysterious monuments on the planet. Nearly 900 giant stone heads scattered across a remote island in the middle of the Pacific. Now, brand new evidence is challenging everything we thought we knew about Easter Island’s awe-inspiring statues – and those who made them. Drawing on the latest science, this authoritative documentary radically rewrites the story of Easter Island.</Description>
    </Programme>
    <Programme>
        <Channel>BBC2</Channel>
        <Name>Elsa the Lioness</Name>
        <Date>2024-06-21</Date>
        <Time>18:00</Time>
        <Description>First transmitted in 1961, David Attenborough visits Joy and George Adamson in Kenya and meets Elsa the lioness and her cubs shortly before Elsa's death. First transmitted in 1961, David Attenborough travels to Meru National Park in Kenya to visit Joy and George Adamson and meet Elsa the lioness and her cubs shortly before Elsa's death.</Description>
    <Programme>
<Programmes>

I would like you to generate data for about 70 programmes with dates ranging from 2024-06-21 to 2024-06-30
and with starting times on the hour or at 15, 30 or 45 minutes past the hours.
Make the programme names and descriptions as realistic as possible.
The genres of some of the programmes should be film.

Such a simple thing saved a lot of programming time. As often seems to be the case when working with Claude, I was surprised at some of the detail in the fictitious schedule. The full database is shown in the GitHub repository in the file TvDb.xml. But, for example, the programme “Good Morning Britain” has been schedule for 06:30am and “The Sky at Night” for 10:30pm.

Final Thoughts

Even though this is a very simple application it made it more clear to me how an AI such as Claude could be programmed to make use of specific locally stored information and combine it with the language, and wider knowledge, that the AI has, resulting in a useful tool.

Being able to precisely describe the expected inputs and outputs makes it easier to work with Claude from a traditional programming language.

AIs, such as Claude, can often be used to help construct parts of the program. In our case it built our TV database, but it could also have been used to write some of the code used to interact with the SDK.

It is worth stressing in the prompts the importance of checking the results before returning them, and the -v flag to get Claude to explain its workings was a useful addition.

Simple Array Manipulation Task

That task involved the following steps - their implementation in Goal is given below:

1. Write a function, outArray that, given a string title and an array of integers as its two parameters, outputs that array as a comma-separated list, preceded by the title, a count of the number of integers in square-brackets and a colon. So, called with the parameters "Initial" and 1 2 3 4, it would return:

   Initial [4]: 1, 2, 3, 4

2. Assign a small array of ints, say, the array: 31 5 7 6 3 2 8 to the variable ints.

3. Use the outArray function to output the ints array with the title “Initial”.

4. Create a new array, sortedInts, from the original ints which contains the elements but sorted into ascending order, and output it with the outArray function using the title “Sorted”.

5. Create an array, oddInts, that contains just those ints from sortedInts that are odd and output it with the title “Odd”.

6. Create an array, argvInts, that has been read from the command line and output it with the title “argvInts”.

7. Create an array, fileInts, that has been read from a file on the filesystem that contains a single line of comma-separated integers and output it with the title “FileInts”.

8. Combine the oddInts, argvInts, and fileInts arrays into a single array, combinedInts, without duplicates and output it with the title “CombinedInts”.

9. Calculate the average of the numbers in the combinedInts array and store it in the scalar average.

10. Split the numbers in the combinedInts array into two distinct arrays, higher and lower such that higher contains those ints that are higher than the value in average and lower contains the remaining ints. Output both higher and lower with `outArray’.

The Goal implementation is:

outArray: {
    [title; ints]
    print "%s[%d]: %s"$(title; #ints; csv ,ints)
}

ints: 31 5 7 6 3 2 8
outArray["Initial"; ints]

sortedInts: ints[<ints]
outArray["Sorted"; sortedInts]

oddInts: sortedInts[&2!sortedInts]
outArray["Odd"; oddInts]

argvInts: "i"$'1_ARGS
outArray["ArgvInts"; argvInts]

fileInts: *"i"$csv 'read "./fileints.csv"
outArray["FileInts"; fileInts]

combinedInts: ?oddInts, argvInts, fileInts
outArray["CombinedInts"; combinedInts]

average: {(+/x)%#x} combinedInts
say "average: %f"$average

higher: {x@&x > average}combinedInts
outArray["Higher";higher]

lower: {x@&~x in higher}[combinedInts]
outArray["Lower";lower]

If we invoke goal with the arguments ‘2 3 4 5’ and if the current directory contains a CSV format file named “fileints.csv” containing the line: 5,55,25,15 we will see the following output.

Initial[7]: 31,5,7,6,3,2,8
Sorted[7]: 2,3,5,6,7,8,31
Odd[4]: 3,5,7,31
ArgvInts[4]: 2,3,4,5
FileInts[4]: 5,55,25,15
CombinedInts[9]: 3,5,7,31,2,4,55,25,15
average: 16.333333
Higher[3]: 31,55,25
Lower[6]: 3,5,7,2,4,15

The OutArray Function

outArray: {
    [title; ints]
    print "%s[%d]: %s"$(title; #ints; csv ,ints)
}

We define outArray to be a function (that is what the curly braces indicate) that takes two parameters title and ints. The last line of the function is its return value. The $ verb, with a format string on its left and one or more values on its right is known as format. It returns a formatted string value in much the same way that sprintf might in a C-like language. This format string expects 3 arguments, a string, an integer and another string: namely, the title, a count of the number of integers in ints and a list of the integers in comma-separated format.

The monadic verb # applied to an array is known as length and gives the length of the array. So #ints is the number of integers in the array.

The monadic verb csv when applied to a generic array returns a string containing the array values in comma-separated format. The monadic verb ,, known as enlist, converts our integer array into a generic array. The monadic verb print prints its argument to standard output. (The monadic verb say, seen later, does the same as print but appends a newline.)

Assignment and Output

We can define an array, ints, and display it with our function:

ints: 31 5 7 6 3 2 8
outArray["Initial"; ints]    

A variable name followed by a colon (:) is known as assign. The variable named on the left is assigned the value of the expression on the right. OutArray is just a variable that has been assigned the value of a lambda (function).

Sort the Array

We can sort the integer array and assign the sorted array to the variable sortedInts.

sortedInts: ints[<ints]
outArray["Sorted"; sortedInts]

The monadic verb <, known as ascend, when applied to an array of integers, ints say, gives an array of the same length in which the value of each element of the new array is an index into the original array.
Picking elements from ints using that array of indexes will result in an array of the elements in the original array arranged in ascending order. (Indexes in Goal start at zero)

So for example:

ints: 31 5 7 6 3 2 8
<ints

gives

5 4 1 3 2 6 0

If we pick the 5th, 4th, 1st, 3rd, 2nd, 6th and 0th elements from ints we get:

2 3 5 6 7 8 31 

So <ints is an array of 7 indexes into ints. We can pick the values at each of these indexes by ‘indexing’ our ints array with the list of indexes:

ints[<int]

is equivalent to:

ints[5 4 1 3 2 6 0]

and results in:

2 3 5 6 7 8 31 

Select the Odd Integers

oddInts: sortedInts[&1=2!sortedInts]
outArray["Odd"; oddInts]

Remainder modulus 2 is obtained with the dyadic ! verb such that 2!sortedInts results in an array of 0s and 1s of the same length as sortedInts but with a 0 in the place of every integer that is even and a 1 in the place of every odd integer. So, given sortedInts (2 3 5 6 7 8 31)

2!sortedInts

gives:

0 1 1 0 1 0 1

In which each position that contains a 1 corresponds to an odd number in sortedInts. The monadic verb &, known as where, when given an array of 0s and 1s returns an array of those indexes that contain a 1. So:

&2!sortedInts

gives:

1 2 4 6

Indexing sortedInts by this array gives us an array of those elements that are odd.

Reading from the Environment

Access to the arguments with which the Goal script was invoked is obtained through the global variable ARGS. If Goal is invoked as

goal intlist.goal 2 3 4 5

then ARGS contains an array of the following string values: "intlist.goal" "2" "3" "4" "5". We would like to discard the first element which contains the script name and convert the remaining elements to an integer array. The dyadic verb _ when presented with an array as its right argument, y say, and an integer as its left argument, x say, is known as drop and will return a new array made from all but the first x elements of y.

argvInts: "i"$'1_ARGS
outArray["ArgvInts"; argvInts]

So

1_ARGS

drops the first element of ARGS and returns the result as a new array. But this is an array of strings and we want an array of integers.

The dyadic adverb ', with a verb on its left and an array on its right is known as each. Each is similar to the map function in many other functional languages - it will apply the verb to each of the elements of the array resulting in a new array.

The monadic verb “i”$ when given a string as its argument is known as parse and, more specifically, the dyadic verb $ with its hardwired left argument of “i” will parse a string into an integer. So the combination of "i"$ and each will convert each string element into an integer element. Given the above value for ARGS

argvInts: "i"$'1_ARGS

will set argvInts to the integer array with value:

2 3 4 5

This assignment could be slightly simplified because "i"$ will operate on an array argument and so, rather than employing the use of each, we could have written the line as:

argvInts: "i"$1_ARGS

But is serves to illustrate the use of each and so has been left in.

Reading from a File

fileInts: *"i"$csv 'read "./fileints.csv"
outArray["FileInts"; fileInts]

We read the contents of the file “./fileints.csv” with:

read "./fileints.csv"

which returns the string:

"5,55,25,15"

We can convert that to an array of strings by using the monadic form of the verb csv which will parse a string into comma-separated components:

csv read "./fileints.csv"

gives:

,"5" "55" "25" "15"

This is potentially a multi-row array containing a row for each line of CSV content in the file. In our case we know that we only have a single line but we still have a 1x4 array of strings. We can convert this to a 1x4 array of integers with:

"i"$csv read "./fileints.csv"

And, finally, we can select the first row using the monadic first verb. So our final assignment looks like this:

fileInts: *"i"$csv 'read "./fileints.csv"

The apostrophe (’) character that was prepended onto the read verb is known as try. Try will cause an early return, with a suitable error message, if read returns an error. For example:

 "open ./fileints.csv: no such file or directory"

Without try control would continue into csv and beyond and the actual error message would be lost.

Combine without Duplicates

To combine three integer arrays into a single array and remove duplicates we make use of the dyadic join verb (,) and the monadic distinct verb (?).

combinedInts: ?oddInts, argvInts, fileInts
outArray["CombinedInts"; combinedInts]

First the arrays are joined into a single array and then the duplicates are removed by applying the distinct verb.

Calculate the Average

average: {(+/x)%#x} combinedInts
say "average: %f"$average

We construct a function to calculate the average value of an array of integers. Unless we declare the names of a function’s arguments they are assumed to be named x, y and z. If you want a function with more than 3 arguments you need to define the names of the arguments as we did for the outArray function above. Our average function looks like this (with spaces added for readability):

{(+/x) % #x}

The expression (+/x) calculates the sum of the integers in x. The expression #x calculates the length of x - the number of elements in x. The verb % performs floating point division. We have to bracket the components that calculate the sum because Goal is right-associative and without the bracketing the function would be parsed as (+/)(x % #x) which is not what we want.

The adverb ‘/’ is known as fold. With a verb on its left it applies the verb between each of the elements of the array on its right. So with the dyadic verb (+), known as add, the expression +/x sums the integers in the array x.

Our function is invoked with combinedInts as its x argument by what is known as “Implicit At Indexing” - When Goal sees two nouns together (in this case a function and an integer array) it assumes that we want to call the function with the array as its argument. Equivalent ways of invoking the function would be by using square brackets (M-expression), using ‘@’ (known as Apply At) or using ‘.’ (known as Apply). So all of the following are equivalent:

average: (+/combinedInts)%#combinedInts
average: {(+/x)%#x} combinedInts
average: {(+/x)%#x}[combinedInts]
average: {(+/x)%#x}@combinedInts
average: {(+/x)%#x}.,combinedInts

Split into Higher and Lower Arrays

higher: {x@&x > average}[combinedInts]
outArray["Higher";higher]

lower: {x@&~x in higher}[combinedInts]
outArray["Lower";lower]

Given our variable average that contains our calculated average and our combinedInts array:

say "average: %f"$average
outArray["CombinedInts"; combinedInts]

gives:

average: 16.333333
CombinedInts[9]: 3,5,7,31,2,4,55,25,15

The expression

combinedInts > average

results in an integer array in which each element of combinedInts which is greater than the average scalar value is replaced with a 1 and all other elements are replaced with a zero.

0 0 0 1 0 0 1 1 0

We can find the indices of these ones using (‘&’) where:

&combinedInts > average

These are the indices into combinedInts at which elements that are greater than average can be found:

3 6 7

Indexing combinedInts using these indexes:

combinedInts@&combinedInts > average

gives:

31 55 25

Tidying it up and placing into a function:

higher: {x@&x > average}combinedInts
outArray["Higher";higher]

gives:

Higher[3]: 31,55,25

To find those elements that are not higher than average we calculate:

combinedInts in higher

which returns:

0 0 0 1 0 0 1 1 0

which is an integer array containing a 1 for each element in combinedInts that is also in higher. But we want those elements that are not in higher.

We can apply not to this array:

~combinedInts in higher

giving:

1 1 1 0 1 1 0 0 1

We can express these 1s as indexes using where:

&~combinedInts in higher

which gives:

0 1 2 4 5 8

Using these to index into combinedInts:

combinedInts@&~combinedInts in higher

gives us those integers in combinedInts that are not in higher:

3 5 7 2 4 15

Wrapping it all up into a function and applying the function to combinedInts:

lower: {x@&~x in higher}[combinedInts]
outArray["Lower";lower]

gives:

Lower[6]: 3,5,7,2,4,15

In Summary

Much to my surprise, I found working with Goal to be more satisfying that working with both Ivy and J.

The reach of Ivy is (quite deliberately) somewhat limited and as a result I found that, when using Ivy, I was unable to perform some of the parts of this task.

At the other end of the spectrum, J is a more complex language to learn, having a more difficult syntax involving digraphs and the concept of rank which needs to be understood in order to handle higher dimensional arrays.

Goal feels as though it sits comfortably in between the two, is slightly more easy on the eye than J, and has much more functionality than Ivy. Goal would be my go-to APL-like language of choice.

The OutArray Function

I chose not to attempt a pure J to Ivy conversion for two reasons. Firstly, I wanted to explore what tools were available in the Ivy language and, secondly, I feared that my rather naive attempts at writing J functions would be exposed and I’d feel the need to re-write the J notebook.

A user-defined function to combine two integers with a comma and a space is useful as a helper function.

# Join a and b with a comma and space.
op a comma b = (text a), ', ' ,(text b)

In Ivy this declares a user-defined operator with the name comma that takes two arguments: a as its left-hand argument and b as its right-hand argument. The expression to right of the equals sign defines the body of the operator - in this case a simple expression.

The binary comma operator will combine its arguments into a single vector.

Elements of a vector are normally printed with a separating space between each element. However, if all of the elements in a vector are characters then, as a special case, they are printed without spaces between them.

To ensure that all of the elements in the vector returned by the comma operator are characters we apply the text operator to the ints a and b.

The same trick is performed in the body of the outArray operator to prevent Ivy from inserting unwanted spaces.

# Output title, count of ints and comma-separated ints.
op title outArray ints =
    title, "[", (text rho ints), "]: ", comma/ ints

The arguments to outArray are title on the left-hand side and ints on the right-hand size. Within the body of the operator definition these are treated as local variables.

rho ints

will give us the length of the ints vector. Applying text to the result will ensure that the overall value is a string.

text rho ints

The final part of the line comma/ ints is using the Ivy Reduce operator (/). Reduce modifies the operator that precedes it so that it applies the operator between each of its arguments. The easiest way to see this is by application to the plus operator. In Ivy +/ 1 2 3 4 is the same as 1+2+3+4. Similarly ’*/ 1 2 3 4’ is the same as 1*2*3*4.

Care needs to be taken to remember that an Ivy expression is parsed from right to left. So -/ 1 2 3 4 is equivalent to 1-2-3-4 which is parsed as 1-(2-(3-4)) and so has the value -2 rather than -8 which would be the value in a language where the expression was parsed from left to right.

Assignment and Output

We can assign a vector of integers to the variable ints like this:

ints = 31 5 7 6 3 2 8

And we can use our outArray operator to display their value:

'Initial' outArray ints
Initial[7]: 31, 5, 7, 6, 3, 2, 8

Sort the List

A vector can be sorted using Ivy’s up (or down) operators. Up is a unary operator which when applied to a vector B, say, returns a vector of the same length as B that contains that ordering of indices which, if used to access the elements of B, would return them in ascending order.

We can apply up to our ints vector:

up ints
6 5 2 4 3 7 1

So, to return the elements of ints in ascending order we would need to take the 6th element of ints followed by the 5th, followed by the 2nd and so on.

Indexing of a vector is performed using the [] notation. Index numbering begins at 1 in Ivy. So the first element of vector B is B[1].

A vector can also be indexed using another vector and this results in a new vector. For example, if B is indexed by the vector 3 1 2 we would obtain a new vector whose elements were made from the third, first and second elements of B, in that order.

So, indexing the ints vector by the result of applying up will give us the sorted values from ints:

sortedInts = ints[up ints]
'Sorted' outArray sortedInts
Sorted[7]: 2, 3, 5, 6, 7, 8, 31

Select the Odd Integers

We can select the odd integers from our vector, and assign them to the variable oddInts, with this:

oddInts = (sortedInts mod 2) sel sortedInts

This makes use of two binary operators: mod and sel. The use of round brackets reminds us that Ivy evaluates a line of input from right to left and so we need the brackets to obtain the desired result. Without the brackets ivy would parse the line as:

oddInts = sortedInts mod (2 sel sortedInts) # wrong

The operator mod when used in A mod B returns the remainder on dividing A by B. Any integer divided by 2 will have a remainder of 0 if it is even and a remainder of 1 if it is odd.

Ivy, being an array processing language, operates on entire vectors and so it makes sense to pass a vector as the left-hand operator of mod the resulting vector being the element-wise application of mod 2. You can think of the right-hand argument 2 as being widened to be a vector of the same length as the left-hand argument but consisting of all 2s.

So, for each element of the vector sortedInts that is odd we will see a 1 in the corresponding position in the result and for each element of sortedInts that is even we will see a 0 in the corresponding position.

sortedInts
2 3 5 6 7 8 31
sortedInts mod 2
0 1 1 0 1 0 1

The sel operator allows us to create a new vector by selecting 0 or more items from each position in a vector. In its simplest form it is used like this:

A sel B

Where A is a vector of the same length as B and contains either the value 0 or the value 1 in each position. Then for each position in A that contains a 1 the value from the corresponding position in B is used in the resulting vector and for each position in A that contains a 0 the value from the corresponding position in B is ignored.

So, if sortedInts contains 7 elements, the expression 1 1 1 0 0 0 0 sel sortedInts would return a vector of length 3 containing the first three elements of sortedInts.

sortedInts
2 3 5 6 7 8 31
1 1 1 0 0 0 0 sel sortedInts
2 3 5

As we saw above, our application of mod 2 resulted in a vector of the same length as sortedInts containing just 0s and 1s and with 1s in the positions that corresponded to odd numbers. So, combining the two and printing the results gives us our vector of odd integers.

oddInts = (sortedInts mod 2) sel sortedInts
'Odd' outArray oddInts
Odd[4]: 3, 5, 7, 31

Reading from the Environment and from a File

Ivy does not have the rich library environment that other APL-like languages have. Nor does it have an each operator that might make it easier to produce an operator that could read CSV formatted strings. So (currently) it would not be possible to reproduce steps six and seven of our task to read CSV input from the environment or filesystem. However, it does have a mechanism in which Ivy statements can be read from a file on disk and so we will read argvInts and fileInts using that method.

To read statements from a disk and continue execution we use the )get command, passing it the name of the file containing the ivy statements that we want to read.

So, if the file argvAndFileInts.ivy contains the following two lines:

argvInts = 33 44 55 66
fileInts = 5 8 10 12

We can read these values in from disk and print them out.

)get "argvAndFileInts.ivy"
'ArgvInts' outArray argvInts
ArgvInts[4]: 33, 44, 55, 66
'FileInts' outArray fileInts
FileInts[4]: 5, 8, 10, 12

Combine without Duplicates

We now need to combine the three vectors oddInts, argvInts and fileInts into a single vector of integers without retaining duplicates. This is made easy for us in Ivy with the unique operator.

combinedInts = unique oddInts, argvInts, fileInts

The unique operator is a unary operator that, when applied to an argument B, will return a new vector that is made by removing all of the duplicate elements from B.

There is a second operator in use in the above assignment - the binary comma operator (,). The comma operator, when applied between vectors A and B, will create a new vector in which the elements of B are appended to the elements of A creating a new vector whose length is the sum of the lengths of A and B.

The right-associative nature of ivy means that first argvInts and fileInts are combined and then the result of that combination is combined with oddInts leaving a single vector that consists of all of the elements of the three vectors (including any duplicate elements).

The application of unique to this combined vector gives us our value for combinedInts.

combinedInts = unique oddInts, argvInts, fileInts
'CombinedInts' outArray combinedInts
CombinedInts[11]: 3, 5, 7, 31, 33, 44, 55, 66, 8, 10, 12

Our three input vectors were each of length 4 but our resulting vector is of length 11. Both fileInts and oddInts contained an element 5 and this has been collapsed into a single element by the application of unique.

Calculate the Average

We need to calculate the average of our combined vector. To do this we need to sum the elements of combinedInts and divide the sum by the number of elements it contains.

Counting the number of elements in a vector is easy, we can do this with the unary operator count.

count combinedInts
11

Ivy has no traditional looping statements but it contains a number of operators that operate along the axes of a vector. In this simple example we have been working with vectors of a single dimension but Ivy can handle multiple dimensions.

The operator / (known as Reduce) would be called an adverb in J because it modifies the operation of a verb. In Ivy, Reduce modifies the behaviour of an operator. In the construction of our outArray operator we used Reduce to modify the behaviour of our comma operator. Here we are going to use Reduce to modify the operation of the ‘+’ operator. The expression

combinedInts 
3 5 7 31 33 44 55 66 8 10 12
+/ combinedInts 
274

is equivalent to laying out the elements of combinedInts with the ‘+’ operator between them and so it calculates the sum of the elements. We can use this to calculate the average of the elements. Our first attempt might be:

average = (+/ combinedInts) / count combinedInts
'Average: ',  text average
Average: 274/11

This reminds us that Ivy uses exact rational arithmetic and so in order to see our average as a floating point number (for comparison with our J calculation) we need to convert part of it to float. We can do this with the unary operator float:

average = (+/ combinedInts) / float count combinedInts
'Average: ',  text average
Average: 24.9090909091

Split into Higher and Lower Vectors

The last part of the program is to split the combinedInts vector into two distinct vectors with one vector containing all the integers that are higher than the average and the other containing the remaining integers.

The binary operator ‘>’, when applied to a vector A and an element B will result in a vector of the same length as A but with a one in each position where the element of A was greater than B and a 0 elsewhere. So evaluating combinedInts > average results in this:

combinedInts 
3 5 7 31 33 44 55 66 8 10 12
combinedInts > average
0 0 0 1 1 1 1 1 0 0 0

We can use this with the sel operator (see above) to extract just those elements that are higher than average using bracketing to force the desired interpretation:

higher = (combinedInts > average) sel combinedInts
'Higher: ', text higher
Higher: 31 33 44 55 66

To calculate the rest we make use of two new operators: in and not.

When applied to vectors A and B, A in B will result in a vector of the same length as A but with a 1 in each position where the element in the corresponding position in A was also in the vector B and a 0 in each position where the element in the corresponding position in A was not found in B.

combinedInts 
3 5 7 31 33 44 55 66 8 10 12
higher
31 33 44 55 66
combinedInts in higher
0 0 0 1 1 1 1 1 0 0 0

This tells us which elements are in higher, but we want to know which elements are not in higher. We can reverse the result using the not operator. The not operator will replace all of the 0s in a vector with 1s and all of the 1s in a vector with 0s:

not combinedInts in higher
1 1 1 0 0 0 0 0 1 1 1

Combining this with the sel operator:

lower = (not combinedInts in higher) sel combinedInts
'Lower: ', text lower
Lower: 3 5 7 8 10 12

In Summary

There were many things that made working with Ivy a really pleasurable experience, and only one or two things that didn’t.

• Ivy’s use of relatively short operator names means that expressions, although not as terse as those of J, are still concise and this keeps the code window small and readable.
• The source code for Ivy, written in Go, is very readable and so it is possible to get an understanding of what is happening behind the scenes. This was not the case with the J source code - at least, I didn’t think so.
• Rob Pike describes Ivy as a plaything and a work in progress. A few people have cloned Ivy and added their own extensions but I really hope that Rob is going to extend the language a little further.
• I found that there were some calculations that I did not feel I could perform in Ivy. This is quite likely as a result of my inexperience with APL-like languages. But, for example, the lack of an ‘each’ iterator stumped me on a few occasions.
• Ivy is described, on its GitHub page as a calculator and it is a very capable calculator. It think it is fair to say that is a gross understatement - Ivy is much more than that.

Summary of Operation

The timesheet software (hereafter referred to as TimeSheet) allows me to report on my activity over time and have the individual activities summarised by project and sub-project in a format that makes it easy for me to see how much work I have done on individual projects, sub-projects within projects, or tasks within sub-projects.

The interpretation of project, sub-project and task is very loose. The important part of the interpretation is that they are considered to be hierarchical such that a task is wholly contained within a sub-project and a sub-project is wholly contained within a project. Sometimes a project might be used to represent a customer and the sub-project might represent a distinct piece of work for that customer.

All is made clear by an example. Consider the following entries made in Outlook between 10:00 and 13:00 on the 1st and 2nd of November.

TimeSheet examines Outlook for calendar entries that occur between a range of dates. These entries are known as events in Outlook.

Only entries whose text matches a given format are considered:

<project> – <sub-project> – <task details>

Project, sub-project and task detail are separated by a single dash character which may be surrounded by whitespace characters. Project and sub-project may only contain letters, digits, an underscore character or a forward slash character. Task details may contain any characters.

Entries with any other format are ignored. So, from the entries above, the only Outlook task that is ignored is the “Early Lunch” task as it does not match the desired format. All of the other tasks in our range of days are considered to be tasks of interest and are collected.

The duration of the task is taken from the duration of the corresponding Outlook event.

TimeSheet is a command line application that runs on macOS and Windows. It writes its output to standard output. Given the data shown in Figure 1 above, Figure 2 shows the output produced by running TimeSheet on that data.

The first line shows the range of dates for which events in Outlook have been read. In this example, TimeSheet has been asked to look at all dates between 1st November and 30th November (inclusive) but, to keep things simple, I have only made entries in Outlook for the days of 1st and 2nd of November.

After that comes a block of output for each distinct project found within those events.

Each block begins with a line containing the name of the project and the total time spent working on that project.

Then, for each week included in the range of days for which TimeSheet has been run, we see a breakdown of the time spent each day on the project (in hours and minutes). The weekly breakdown is arranged to begin on a Monday. In this example our data for the 1st and 2nd of November would appear in cells 3 and 4 of the row for the week beginning 30th October. We can see that we worked for 2 hours on Project X on the first of November and that this was the only week (so far) in the month that we have worked on ProjectX.

Next, for the project, we see a block of lines showing the total time that we have spent on each sub-project within the project. We see that, for ProjectX, we have spent 1 hour 30 minutes on documentation and 30 minutes on testing. This sub-project summary is only for sub-projects within the main project. Were there to also be a Doc sub-project within, say, the CompanyY project then its Doc sub-project would be totalled separately. So it is OK to have sub-projects with the same title within different projects and we will never double count them.

After the sub-projects block comes the task block. The task block is more than just a list of tasks; it is a summary of the distinct tasks within the sub-projects. So if multiple entries exist with the same project, sub-project and task detail they appear as just one line here and their total duration is summarised. Working on a task, taking a break, and then working on the same task at some point later on will result in just a single line in this task block with the duration being the total time that has been spent on that task in the (project and sub-project and) date range. Each line in the task block begins with the sub-project to which the task belongs. This allows us to distinguish between tasks that have the same task detail within different sub-projects of a project.

In summary, we use a very simple input format within an event in Outlook and we generate a textual output that enables us to see at a glance a summary of what we have been doing at the project, sub-project, task and day level.

Complications

There are two complications that didn’t exist in the original COM-based F# timesheet program:

1. Outlook is now part of Microsoft 365 and so the Outlook event data lives within Microsoft’s servers rather than on the local machine.
2. To be able to obtain access to the data on Microsoft’s servers some kind of authentication will need to be done.

Outlook event data can be read using the Microsoft Graph API. But, because we want to run TimeSheet on our local machine, we will need to be able to authenticate first and so I will look at the authentication mechanism first in this document. Authentication is the trickiest part of this interaction with Microsoft’s servers but its complexity is handled by the Microsoft Graph SDK for Go and the Kiota Azure Identity authentication provider library for Go.

Once authentication has been configured we can obtain a client for calling the Microsoft Graph service. The graph service client allows us to query calendar events between a range of dates and once we can do that we then have our event data available to play with.

Authentication with Microsoft Entra ID

The TimeSheet Application

The TimeSheet application is written in Go. There are four services with a very simple relationship between them. The main module reads the command line flags and the locally stored secrets and, through the timesheet service, wires up the other three services. It then uses them to generate the textual result and writes it to standard output.

flowchart LR
  m(main) --> t(timesheet)
  t(timesheet) --> g(graph)
  t(timesheet) --> c(calendar)
  t(timesheet) --> d(dump)

The parameters and secrets read by the main module include the range of dates of interest, the Outlook username, and the authentication details. The authentication details are passed to the graph service which uses them to obtain a client that can access the Microsoft Graph service. The timesheet service calls the graph service with username and range of dates to obtain a list of the relevant events that happened for the given user between the two dates. The timesheet service passes this list of tasks to the calendar service which summarises and aggregates event times, returning them as a list of Projects. These are then passed to the dump service to convert them into our desired output format for output on standard output.

There are four steps required to get to the point where we can retrieve Outlook event data:

1. Register this app with the Microsoft Identity Platform: We have already done this (see above).
2. Create an AuthenticationProvider object.
3. Get a Graph Service Client and Adapter object.

go get github.com/microsoftgraph/msgraph-sdk-go
go get github.com/microsoft/kiota-authentication-azure-go

import (
    azidentity "github.com/Azure/azure-sdk-for-go/sdk/azidentity"
    msgraphsdk "github.com/microsoftgraph/msgraph-sdk-go"
)
...
type Auth struct {
    TenantId     string
    ClientId     string
    ClientSecret string
}
...
func NewGraphSvc(auth domain.Auth) domain.GraphSvc {
    cred, err := azidentity.NewClientSecretCredential(
        auth.TenantId,
        auth.ClientId,
        auth.ClientSecret,
        nil,
    )
    if err != nil {
        fmt.Printf("Error creating credentials: %v\n", err)
        return graphSvc{auth: auth, client: nil}
    }
    scopes := []string{"https://graph.microsoft.com/.default"}
    client, err := msgraphsdk.NewGraphServiceClientWithCredentials(cred, scopes)
    if err != nil {
        fmt.Printf("Error creating graph client: %v\n", err)
        return graphSvc{auth: auth, client: nil}
    }
    return graphSvc{
        auth:   auth,
        client: client,
    }
}

export UserName="john.bates@oldgang.net",
export TenantId="g3xxxxxx-9999-xxxx-xxxx-1xxxx1x2xxxx",
export ClientId="xpxxxxf1-9999-4xxe-xxxx-dxxxx3x4x9x5",
export ClientSecret="kljkliexxxxxxxllxxxxxlxxxxxxCP(sGbYFwcWP"

type GraphSvc interface {
    Read(userName string, fromDate time.Time, toDate time.Time) ([]Task, error)
}

func (svc graphSvc) Read(userName string, fromDate time.Time, toDate time.Time) ([]domain.Task, error) {

    if svc.client == nil {
        return []domain.Task{}, nil
    }
    allusers, err := svc.client.Users().Get(context.Background(), nil)
    if err != nil {
        return []domain.Task{}, err
    }
    if allusers == nil || allusers.GetValue() == nil {
        return []domain.Task{}, nil
    }
    var targetUser models.Userable
    for _, user := range allusers.GetValue() {
        if *user.GetUserPrincipalName() == userName {
            targetUser = user
        }
    }
    if targetUser == nil {
        return []domain.Task{}, nil
    }
    ...

    // Got the user. Now get the tasks for that user.
    // Microsoft graph stores datetimes in UTC. So convert our range to UTC before filtering.
    start := fromDate.UTC().Format("2006-01-02T15:04:05.0000000")
    end := toDate.UTC().Format("2006-01-02T15:04:05.0000000")
    filter := fmt.Sprintf("start/DateTime ge '%s' and start/DateTime le '%s' and IsAllDay eq false", start, end)
    query := users.ItemCalendarEventsRequestBuilderGetQueryParameters{
        Select: []string{"subject", "start", "end"},
        Filter: &filter,
        Top:    &[]int32{999}[0],
    }
    options := users.ItemCalendarEventsRequestBuilderGetRequestConfiguration{
        QueryParameters: &query,
    }
    events, err := svc.client.Users().ByUserId(*targetUser.GetId()).Calendar().Events().Get(context.Background(), &options)
    if err != nil {
        return []domain.Task{}, err
    }
    if events == nil || events.GetValue() == nil {
        return []domain.Task{}, nil
    }
    ...

    tasks := []domain.Task{}
    for _, ev := range events.GetValue() {
        if ev == nil || ev.GetSubject() == nil {
            continue
        }

        // proj (- group) - description
        var pat = `^\s*([\w/]+)(?:\s*-\s*([\w/]+))?\s*-\s*(.*)`

        matches := regexp.MustCompile(pat).FindStringSubmatch(*ev.GetSubject())
        if matches == nil || len(matches) != 4 { // Entire expression plus each subexpression.
            continue
        }
        proj := matches[1]
        group := matches[2]
        desc := matches[3]
        _start := ev.GetStart()
        _end := ev.GetEnd()
        start, err := time.Parse("2006-01-02T15:04:05.0000000", *_start.GetDateTime())
        if err != nil {
            return []domain.Task{}, fmt.Errorf("failed to parse start time: %v", err)
        }
        end, err := time.Parse("2006-01-02T15:04:05.0000000", *_end.GetDateTime())
        if err != nil {
            return []domain.Task{}, fmt.Errorf("failed to parse end time: %v", err)
        }
        // Convert back from UTC to local time.
        start = start.Local()
        end = end.Local()
        duration := end.Sub(start)
        tasks = append(tasks, domain.Task{
            Project:  proj,
            Group:    group,
            Desc:     desc,
            Start:    start,
            Duration: duration,
        })
    }
    return tasks, nil
}

type CalendarSvc interface {
    Aggregate(tasks []Task) []*Project
}

type DumpSvc interface {
    Projects(projects []*Project) []string
}

Usage

There are only three things that the TimeSheet application needs to know to produce some output: a starting date, an ending date and a user name.

The username, which is often the same for each invocation, is most usefully stored in the .envrc file. But an alternative username may be provided with the ‘-user’ flag.

If run without any arguments the TimeSheet application will assume a date range that spans the start to the end (inclusive) of the current month.

The ‘-n’ flag can be used to request a timesheet for ‘n’ months back from the current month and the date range that spans that month will be used. So, for example, running:

./TimeSheet -n 1

will output a timesheet summary for the previous month.

The ‘-from’ and ‘-to’ flags may be used to provide alternative starting and ending dates. The dates must be provided in the YYYY-MM-DD format and may be used in conjunction with the ‘-n’ flag to modify the dates that the ’-n- flag would imply.

So, for example, the following would only report on events in my calendar that happened on the first day of November 2023.

./TimeSheet -from '2023-11-01' -to '2023-11-01' -user 'john.bates@oldgang.net'

Because a time sheet can only be constructed after the event, and is most usefully constructed as soon after the event as possible, the most common invocation is:

./TimeSheet -n 1

to summarise last month’s timesheet.

Summary of Operation

The timesheet software (hereafter referred to as TimeSheet) allows me to report on my activity over time and have the individual activities summarised by project and sub-project in a format that makes it easy for me to see how much work I have done on individual projects, sub-projects within projects, or tasks within sub-projects.

The interpretation of project, sub-project and task is very loose. The important part of the interpretation is that they are considered to be hierarchical such that a task is wholly contained within a sub-project and a sub-project is wholly contained within a project. Sometimes a project might be used to represent a customer and the sub-project might represent a distinct piece of work for that customer.

All is made clear by an example. Consider the following entries made in Outlook between 10:00 and 13:00 on the 1st and 2nd of November.

TimeSheet examines Outlook for calendar entries that occur between a range of dates. These entries are known as events in Outlook.

Only entries whose text matches a given format are considered:

<project> – <sub-project> – <task details>

Project, sub-project and task detail are separated by a single dash character which may be surrounded by whitespace characters. Project and sub-project may only contain letters, digits, an underscore character or a forward slash character. Task details may contain any characters.

Entries with any other format are ignored. So, from the entries above, the only Outlook task that is ignored is the “Early Lunch” task as it does not match the desired format. All of the other tasks in our range of days are considered to be tasks of interest and are collected.

The duration of the task is taken from the duration of the corresponding Outlook event.

TimeSheet is a command line application that runs on macOS and Windows. It writes its output to standard output. Given the data shown in Figure 1 above, Figure 2 shows the output produced by running TimeSheet on that data.

The first line shows the range of dates for which events in Outlook have been read. In this example, TimeSheet has been asked to look at all dates between 1st November and 30th November (inclusive) but, to keep things simple, I have only made entries in Outlook for the days of 1st and 2nd of November.

After that comes a block of output for each distinct project found within those events.

Each block begins with a line containing the name of the project and the total time spent working on that project.

Then, for each week included in the range of days for which TimeSheet has been run, we see a breakdown of the time spent each day on the project (in hours and minutes). The weekly breakdown is arranged to begin on a Monday. In this example our data for the 1st and 2nd of November would appear in cells 3 and 4 of the row for the week beginning 30th October. We can see that we worked for 2 hours on Project X on the first of November and that this was the only week (so far) in the month that we have worked on ProjectX.

Next, for the project, we see a block of lines showing the total time that we have spent on each sub-project within the project. We see that, for ProjectX, we have spent 1 hour 30 minutes on documentation and 30 minutes on testing. This sub-project summary is only for sub-projects within the main project. Were there to also be a Doc sub-project within, say, the CompanyY project then its Doc sub-project would be totalled separately. So it is OK to have sub-projects with the same title within different projects and we will never double count them.

After the sub-projects block comes the task block. The task block is more than just a list of tasks; it is a summary of the distinct tasks within the sub-projects. So if multiple entries exist with the same project, sub-project and task detail they appear as just one line here and their total duration is summarised. Working on a task, taking a break, and then working on the same task at some point later on will result in just a single line in this task block with the duration being the total time that has been spent on that task in the (project and sub-project and) date range. Each line in the task block begins with the sub-project to which the task belongs. This allows us to distinguish between tasks that have the same task detail within different sub-projects of a project.

In summary, we use a very simple input format within an event in Outlook and we generate a textual output that enables us to see at a glance a summary of what we have been doing at the project, sub-project, task and day level.

Complications

There are two complications that didn’t exist in the original COM-based F# timesheet program:

1. Outlook is now part of Microsoft 365 and so the Outlook event data lives within Microsoft’s servers rather than on the local machine.
2. To be able to obtain access to the data on Microsoft’s servers some kind of authentication will need to be done.

Outlook event data can be read using the Microsoft Graph API. But, because we want to run TimeSheet on our local machine, we will need to be able to authenticate first and so I will look at the authentication mechanism first in this document. Authentication is the trickiest part of this interaction with Microsoft’s servers but its complexity is handled by components of .NET which, thankfully, keeps the number of lines of user code required to be relatively low.

Once authentication has been configured we can obtain a client for calling the Microsoft Graph service. The graph service client allows us to query calendar events between a range of dates and once we can do that we then have our event data available to play with.

Authentication with Microsoft Entra ID

The TimeSheet Application

The TimeSheet application is written in C# for the Microsoft .NET Core platform. There are five modules with a very simple relationship between them. The main module wires up the other four modules and outputs the result to standard output.

flowchart LR
  p(param) --> g(graph)
  g(graph) --> c(calendar)
  c(calendar) --> d(dumper)

  p --> m(main)
  d --> m(main)

The param module gathers the parameters that will be used to query the calendar. These include the range of dates of interest and the authentication details. These are passed to the graph module which uses them to obtain a client that can access the Microsoft Graph service. The calendar module uses graph service client to query the service and return a set of calendar entries which it then groups and summarises by project. The dumper module is then responsible for formatting this summary and returning it to the main module.

Code for the main module looks like this:

class Program
{
    static async Task Main(string[] args)
    {
        // Wire up the components.
        var param = parameter.Reader.New(args);
        var (graph, error) = msgraph.Client.New(param.Auth);
        if(graph == null) { // error will be non-null
            Console.Error.WriteLine(error);
            System.Environment.Exit(1);
        }
        var calendar = task.Reader.New(graph);
        var dumper = dump.Dump.New(calendar);

        // Read the parameters
        var userPrincipalName = param.UserPrincipalName;
        var dateFrom = param.DateFrom;
        var dateTo = param.DateTo;
        if (userPrincipalName == null){
            Console.Error.WriteLine("A value for UserPrincipalName is required");
            System.Environment.Exit(1);
        }
        if (dateFrom == null || dateTo == null){
           Console.Error.WriteLine("A value for both DateFrom and DateTo is required"); 
            System.Environment.Exit(1);
        }

        // Output the results
        var output = await dumper.summary(userPrincipalName, dateFrom.Value, dateTo.Value);
        foreach(var s in output){
            Console.WriteLine(s);
        }
    }
}

public interface IDumper {
    public Task<List<string>> summary(string userPrincipalName, DateTime fromDate, DateTime toDate, bool satweek=false);
}

public static (IClient? client, string? error) New(azure.Auth auth)
{

    TokenAcquirerFactory tokenAcquirerFactory = TokenAcquirerFactory.GetDefaultInstance();
    IServiceCollection services = tokenAcquirerFactory.Services;
    services
        .AddMicrosoftGraph();

    // Ensure that the tokenAcquirerFactory has the required authentication configuration.
    var C = tokenAcquirerFactory.Configuration;
    C["AzureAd:Instance"] = auth.Instance;
    C["AzureAd:TenantId"] = auth.TenantId;
    C["AzureAd:ClientId"] = auth.ClientId;
    C["AzureAd:ClientCredentials:0:SourceType"] = "ClientSecret";
    C["AzureAd:ClientCredentials:0:ClientSecret"] = auth.ClientSecret;

    var serviceProvider = tokenAcquirerFactory.Build();
    if(serviceProvider == null)
        return (null, "Could not create service provider");
    var graphclient = serviceProvider.GetService<GraphServiceClient>();
    if(graphclient == null)
        return(null, "Could not create graph service client");
    return (new Client(graphclient), null);
}

var builder = new ConfigurationBuilder()
    .SetBasePath(Directory.GetCurrentDirectory())
    .AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
    .AddJsonFile("env.private", optional:false, reloadOnChange: false)
    .AddEnvironmentVariables();
IConfigurationRoot configuration = builder.Build();

{
    // These are made up values.
    "Instance": "https://login.microsoftonline.com/",
    "TenantId": "f3xxxxxx-9999-xxxx-xxxx-1xxxx8x3xxxx",
    "ClientId": "xbxxxxf1-9999-4xxe-xxxx-dxxxx5x4x9x5",
    "ClientSecret": "kljkliexxxxxxxllxxxxxlxxxxxxCY7sG7nfwcWP"
}

public interface IClient
{
    Task<List<task.Task>> TimesheetItems(string userPrincipalName, DateTime dateFrom, DateTime dateTo);
}

public async Task<List<task.Task>> TimesheetItems(string userPrincipalName, DateTime dateFrom, DateTime dateTo){
        var tasks = new List<task.Task>();
        var users = await graphclient.Users.GetAsync(
            r => r.Options.WithAppOnly()
        );
        if(users == null || users.Value == null)
            return tasks;
        User? targetUser = null;
        foreach (var user in users.Value) {
            if (user.UserPrincipalName == userPrincipalName)
                targetUser = user;
        }
        if (targetUser != null) {
            var events = await graphclient
                .Users[$"{targetUser.Id}"]
                .Calendar.Events.GetAsync(r => {
                    r.QueryParameters.Select = new []{"subject", "start", "end"};
                    // Microsoft graph stores datetimes in UTC. So convert our range to UTC.
                    var start = dateFrom.ToUniversalTime().ToString("yyyy-MM-ddTHH:mm:ss.0000000");
                    var end = dateTo.ToUniversalTime().ToString("yyyy-MM-ddTHH:mm:ss.0000000");
                    r.QueryParameters.Filter = $"start/DateTime ge '{start}' and start/DateTime le '{end}' and IsAllDay eq false";
                    r.QueryParameters.Top = 999;
                    r.Options.WithAppOnly();
                });
            if (events == null || events.Value == null)
                return tasks;
            var eventList = events.Value.ToList();
            foreach (var ev in eventList)
            {
                if(ev == null || ev.Subject == null)
                    continue;
                // proj (- group) - description
                var pat = @"^\s*([\w/]+)(?:\s*-\s*([\w/]+))?\s*-\s*(.*)";
                var m = Regex.Match(ev.Subject, pat) ;
                if (m.Success){
                    string proj = m.Groups[1].Value;
                    string group = m.Groups[2].Value;
                    string desc = m.Groups[3].Value;
                    // Convert back to local time.
                    DateTime start = ev.Start.ToDateTime().ToLocalTime();
                    DateTime end = ev.End.ToDateTime().ToLocalTime();
                    TimeSpan duration = end.Subtract(start); 
                    tasks.Add (new task.Task(proj, group, desc, start, duration));
                }
            }
            return tasks;
        }
        return tasks;
    }

Usage

There are only really three parameters that need to be passed to the TimeSheet application to obtain some output: a starting date, an ending date and a user name.

The param module handles the gathering of these values from: the “appsettings.json” file, the “env.private” file or the environment.

Typically the “env.private” file will contain the authentication details (as shown above) and the “appsettings.json” file will hold the UserPrincipalName and so will look like this:

{
    "UserPrincipalName": "john.bates@oldgang.net"
}

If run without any arguments the TimeSheet application will assume a date range that spans the start to the end (inclusive) of the current month. If given a single numeric argument this is interpreted as a number of months back from the current month and the date range that spans that month will be used. So, for example, running:

    ./TimeSheet 1

will output a timesheet summary for the entire previous month.

The “appsettings.json” file may also contain values for DateFrom and DateTo to overwrite the default values. So, for example, an “appsetting.json” file containing the following would only report on events in my calendar that happened on the first day of November 2023.

{
    "UserPrincipalName": "john.bates@oldgang.net",
    "DateFrom": "1 Nov 2023",
    "DateTo": "2023-11-01"
}

As demonstrated in the above example, any suitably unambiguous DateTime format that can be parsed by the .NET DateTime parser can be used to represent the date range.

These three parameters can also be provided from the environment as environment variables, and on macOS may be prefixed on the command line. For example:

    DateFrom="1 Nov 2023" DateTo="4 Nov 2023" ./TimeSheet

In spite of all that flexibility, because a time sheet can only be constructed after the event and is most usefully constructed as soon after the event as possible, the most common invocation is:

    ./TimeSheet 1

to summarise last month’s timesheet.

Responsibility

Consider this simple main function that calls ‘readWorkbook’ to read a workbook from disk into a structure. The function returns two values, a populated workbook structure and a Go error. This is ideomatic go - if a function needs to return an error it is normally passed back as the last returned value.

A nil error is conventionally interpreted to mean that the function worked and so the code will normally check first for a nil error before using the other returned values. In our example, if we know that there is no error we can safely use the other values returned by our function: wb.sheets and wb.name.

package main

import (
    "fmt"
    "io"
    "os"
    "strings"
)

func main() {
    wbPath := "./recipies.xlsx"
    wb, err := readWorkbook(wbPath)
    if err != nil {
        fmt.Printf("Error in '%s': %v\n", wbPath, err)
    } else {
        fmt.Printf("Read %d sheets from workbook '%s'\n", len(wb.sheets), wb.name)
    }
}

But there is a problem. If recipies.xlsx does not exist in the current directory we get:

Error in './recipies.xlsx': open ./recipies.xlsx: no such file or directory

The caller has no way of knowing that the called function is going to return the name of the file in ‘err’. Or, rather, in all of its potential error messages.

Our added message is also somewhat misleading. We say that there was an error ‘in’ the file, but the error in this case was in opening the file and had nothing to do with the contents of this file.

All of this suggests that we might be better off pushing the reporting of the filename up into the readWorkbook function.

This is the original definition of readWorkbook.

type workbook struct {
    name   string
    sheets []*sheet
}

type sheet struct {
    name string
    rows []*row
}

type row struct {
    cells []string
}

func readWorkbook(name string) (*workbook, error) {
    file, err := os.Open(name)
    if err != nil {
        return nil, err
    }
    bytes, err := io.ReadAll(file)
    if err != nil {
        return nil, err
    }
    sheets, err := readSheets(bytes)
    if err != nil {
        return nil, err
    }
    return &workbook{name, sheets}, nil
}

We can see that the library function os.Open is able to return the filename in its error message but io.ReadAll and readSheets will know nothing about the filename.

If we push the responsibility for adding the workbook name up into the readWorkbook function we get:

func main() {
    wbPath := "./recipies.xlsx"
    wb, err := readWorkbook(wbPath)
    if err != nil {
        fmt.Println(err)
    } else {
        fmt.Printf("Read %d sheets from workbook '%s'\n", len(wb.sheets), wb.name)
    }
}
...
func readWorkbook(name string) (*workbook, error) {
    fail := func(op string, err error) (*workbook, error) {
        return nil, fmt.Errorf("read work book (%s): %w", op, err)
    }
    file, err := os.Open(name)
    if err != nil {
        return fail("open", err)
    }
    bytes, err := io.ReadAll(file)
    if err != nil {
        return fail("read "+name, err)
    }
    sheets, err := readSheets(bytes)
    if err != nil {
        return fail("sheets "+name, err)
    }
    return &workbook{name, sheets}, nil
}

Which, assuming that readSheets can return us a “bad format” error message, will give us error messages like this:

read work book (open): open ./recipies.xlsx: no such file or directory
read work book (sheets ./recipies.xlsx): bad format

These don’t repeat the filename but do allow us to determine the location and kind of the error as well as passing the low level error detail back.

The ‘%w’ verb in the format string passed to fmt.Errorf in fail causes the original error to be wrapped inside the new error such that it can be unwrapped when the chain of errors is displayed later.

This change has resulted in a slightly messier readWorkbook function but our calling function has been simplified and, by pushing the error message generation as far up the stack as we can, we have eliminated the chance of duplicating information in the error message, simplified the calling code and so reduced the work required at other sites that might call readWorkbook.

In the article Error handling and Go on the Go Blog Andrew Gerrand writes that

“It is the error implementation’s responsibility to summarize the context.”

I interpret that, in this context, as meaning that we should aim to construct the error message as close to the cause of the error as is possible whilst still being able to retain as much of the context as it is required to keep.

Traceability

The Google Style Guide for Go says that;

Error strings should not be capitalized (unless beginning with an exported name, a proper noun or an acronym) and should not end with punctuation. This is because error strings usually appear within other context before being printed to the user.

This context is what helps the reader of the error message understand more about the route through the code that arrived at the error.

From our example error message (above):

read work book (open): open ./recipies.xlsx: no such file or directory

As we read from left to right, we home in on the source of the actual error. Here, we might infer that in attempting to read a workbook, the open operation of ./recipies.xlsx failed with a no such file or directory error from the operating system.

To a user of the software this might be sufficient to help them work out how to resolve the problem. However, a software developer will usually want more traceability. The Go language makes it easy to construct error messages that carry much more detail, but one nice package that does this for us is the errgo package developed by Roger Peppe.

Errgo will trace the chain of error returns from its source to the function in which the details of the error are presented. It does this by wrapping the original error with source file and line number information at each point that the error gets returned by a function. This is best seen by example.

Consider the rest of the workbook example, now modified to use errgo. The fail function in readWorkbook now uses errors.Notef to wrap the underlying error.

func readWorkbook(name string) (*workbook, error) {
    fail := func(op string, err error) (*workbook, error) {
        return nil, errors.Notef(err, nil, "read work book (%s)", op)
    }
    file, err := os.Open(name)
    if err != nil {
        return fail("open", err)
    }
    bytes, err := io.ReadAll(file)
    if err != nil {
        return fail("read "+name, err)
    }
    sheets, err := readSheets(bytes)
    if err != nil {
        return fail("sheets "+name, err)
    }
    return &workbook{name, sheets}, nil
}

func readSheets(bytes []byte) ([]*sheet, error) {
    sr, err := NewSheetReader(bytes)
    if err != nil {
        return nil, errors.Wrap(err)
    }
    sheets := []*sheet{}
    for name, bytes := range sr {
        rows, err := readRows(bytes)
        if err != nil {
            return nil, errors.Wrap(err)
        }
        sheets = append(sheets, &sheet{name, rows})
    }
    return sheets, nil
}

func readRows(bytes []byte) ([]*row, error) {
    rr, err := NewRowReader(bytes)
    if err != nil {
        return nil, errors.Wrap(err)
    }
    rows := []*row{}
    for _, bytes := range rr {
        cells, err := readCells(bytes)
        if err != nil {
            return nil, errors.Wrap(err)
        }
        rows = append(rows, &row{cells})
    }
    return rows, nil
}

func readCells(bytes []byte) ([]string, error) {
    cr, err := NewCellReader(bytes)
    if err != nil {
        return nil, errors.Wrap(err)
    }
    cells := []string{}
    for _, s := range cr {
        cells = append(cells, s)
    }
    return cells, nil
}

// NewSheetReader splits the 'bytes' that make up a workbook
// into a map of bytes that make up each sheet indexed by sheet name.
// This implementation hardwires the worksheet.
func NewSheetReader(bytes []byte) (map[string][]byte, error) {
    sheets := map[string][]byte{
        "Sheet1": []byte("A1,B1/A2,B2,C2/A3"),
        "Sheet2": []byte("A1"),
    }
    return sheets, nil
}

// NewRowReader splits the 'bytes' that make up a sheet
// into a slice of bytes that make up each row.
func NewRowReader(bytes []byte) ([][]byte, error) {
    rowstrings := strings.Split(string(bytes), "/")
    rows := [][]byte{}
    for _, row := range rowstrings {
        rows = append(rows, []byte(row))
    }
    if true { // force an error here
        return nil, errors.New("bad format row")
    }
    return rows, nil
}

// NewCellReader splits the 'bytes' that make up a row
// into a slice of strings that make up each row.
func NewCellReader(bytes []byte) ([]string, error) {
    return strings.Split(string(bytes), ","), nil
}

The readSheets, readRows and readCells functions now call errors.Wrap with an error before passing the result back down the call stack. The effect of this is to have information about the source code file and line number added to the error as it is passed down through each function.

For the purposes of generating an error message we have introduced a fictional “bad format row” error in the NewRowReader function.

If in the code where we display the error message we use

fmt.Printf("%#v", err)

in place of:

fmt.Println(err)

Then rather than our simple error message of:

read work book (sheets ./recipies.xlsx): bad format row

instead we see:

[
        {/Users/john/Projects/GoErrors/main.go:44: read work book (sheets ./recipies.xlsx)}
        {/Users/john/Projects/GoErrors/main.go:71: }
        {/Users/john/Projects/GoErrors/main.go:81: }
        {/Users/john/Projects/GoErrors/main.go:126: bad format row}
]

Which gives us a traceable route through our code starting at line 44 in main.go and progressing through lines 71 and 81 to line 126 which is where the error was generated.

Errgo maintains both the normal error message and a detailed stack trace which can be of great diagnostic use to a developer. The programmer needs to devise a mechanism that prevents a user from seeing the detailed message but which allows the developer access - typically this might be through a log file.

Fragile Dependencies

The fmt.Errorf function from the fmt package in the Go standard library can be used to create a new error that wraps an existing error:

fmt.Errorf("read work book (%s): %w", op, err)

When we do this the format of the new error message is controlled by the format string passed to fmt.Errorf in its first argument. Typically, perhaps ideomatically, the message from the underlying error follows the new error message by a colon and a space, as above.

However, the action of wrapping an error does more than just augment an error message, it also makes the actual underlying error available to any function receiving the new error.

Suppose we make a call to os.Open to open an operating system file. In checking for an error we might either choose to return the error we received from os.Open or we might choose to wrap it in another message and return the wrapped error, as here:

file, err := os.Open(name)
if err != nil {
    err = fmt.Errorf("open file: %w", op, err)
    return err
}

In this situation the calling code has access to the original error, including the name of the file that we attempted to open.

The system library function errors.As can be used to interrogate the stack of wrapped errors to see if it contains errors of a particular kind. For example, if an error occurs in a call to the system library function os.Open then the error that is returned will be of type *os.PathError. No matter how deep the stack of wrapped errors, a call to errors.As can be used to extract the PathError and its values.

var perr *fs.PathError
if errors.As(err, &perr) {
    fmt.Println("err", perr.Path)
}

This might not always be desirable. For example, if you are writing an API you will not want to leak what might be seen as implementation detail out into errors that users of the API might choose to rely on.

The errgo package takes a tougher stance on the construction of the wrapped error stack. If an error is wrapped by the errgo package using, say, either errors.Notef or errors.Wrap as above, then calls to the system library errors.Is and errors.As functions will always return false and so no such interrogation will be possible.

If a function wishes to return information about an error and its cause then it has to be quite specific about how it constructs that error. It can make use of the second argument to errors.Notef to specify which types of error can be exposed as a cause of the error, for example:

fail := func(op string, err error) (*workbook, error) {
    return nil, errors.Notef(
        err,
        errors.Is(fs.ErrNotExist),
        "read work book (%s)",
        op,
    )
}

Alternatively it can make use of the errors.Becausef function to explicitly supply the cause of the error (in the second argument to errors.Becausef).

fail := func(op string, err error) (*workbook, error) {
    return nil, errors.Becausef(
        err,
        err,
        "read work book (%s)",
        op,
    )
}

In this example, as well as wrapping the underlying error and adding our own message, we have supplied the underlying error as the cause of the error. By doing this, we have allowed code lower down the call stack to examine the cause of the error by calling the errors.Cause function.

wb, err := readWorkbook(wbPath)
if err != nil {
    cause := errors.Cause(err)
    if cause != err {
        // do something with the cause.
    }
}

If there has been no explicit cause assigned to the error then errors.Cause(err) will return its argument, otherwise it will return the cause of the error. Errgo allows us to both construct an error chain and also to preserve the original cause of an error.

Only the errors.Note, errors.Notef, errors.Because and errors.Becausef functions preserve the error cause - Errors.Wrap does not preserve the original cause of an error. This arrangement makes it less easy to accidentally pass information down the error chain about an implementation detail that code lower down might become dependent on.

Grammar

It makes sense to define a little (simplified) J grammar at this point. What we might commonly call a statement in other languages is referred to in J as a sentence. The executable bits of a sentence are called fragments. A noun holds data. A verb operates on one or two nouns to produce another noun. An adverb operates on a verb to produce another verb.

Evaluation of a sentence is the right-to-left evaluation of the fragments of the sentence, with the result of evaluating a fragment replacing the fragment itself and being passed as an operand to the next fragment.

We saw, above, an example of an adverb / which modified the + verb to produce a new verb +/ which would sum the argument on its right.

A Simple Program

I decided that it was time to write a simple J program, or, rather, to attempt a challenge of sorts. I had been inspired by Rob Pike’s Ivy - an APL-like calculator, and in particular, by Russ Cox’s YouTube Videos of his solutions to the first fourteen days of the Advent of Code 2021 challenges.

My program was going to be somewhat simpler though. I wanted to see how J would compare with a .NET language like F# for simple list manipulation. So I decided on a simple list of tasks, starting with a function to display the contents of a list.

outArray =: 4 : 0
  nums =: (' ' ; ', ') stringreplace (": y)
  ; x;'[';(":#y);']: '; nums
)

2 * 3 + 1
8

nums =: (' ' ; ', ') stringreplace (": y)

1 2 3

nums =: (' ' ; ', ') stringreplace '1 2 3'

1, 2, 3

; x;'[';(":#y);']: '; nums

ints =: 31 5 7 6 3 2 8
'Initial' outArray ints

Initial[7]: 31, 5, 7, 6, 3, 2, 8

/: 5 3 4

1 2 0

sortedInts =: ints /: ints
'Sorted' outArray sortedInts

Sorted[7]: 2, 3, 5, 6, 7, 8, 31

3 1 0 2 # 6 7 8 9

6 6 6 7 9 9

oddInts =: (2 | sortedInts) # sortedInts
'Odd' outArray oddInts

Odd[4]: 3, 5, 7, 31

ourArgv =: 'programName';'33';'44';'55';'66'
argvInts =: ". }. > ourArgv
'ArgvInts' outArray argvInts

ArgvInts[4]: 33, 44, 55, 66

F =: '/Users/john/j903-user/projects/intarray/fileints.csv'
fileInts =: ".;._1 ',', }: freads F
'FileInts' outArray fileInts

FileInts[4]: 5, 8, 10, 12

combinedInts =: ~. oddInts, argvInts, fileInts 
'CombinedInts' outArray combinedInts

CombinedInts[11]: 3, 5, 7, 31, 33, 44, 55, 66, 8, 10, 12

(+/ % #) combinedInts

(+/ combinedInts) % (# combinedInts)

average =: (+/ % #) combinedInts
'Average' outArray average

Average[1]: 24.9091

higher =: ((>&average) combinedInts) # combinedInts
lower =: combinedInts -. higher
'Higher' outArray higher
'Lower' outArray lower

Higher[5]: 31, 33, 44, 55, 66
Lower[6]: 3, 5, 7, 8, 10, 12

In Summary

It is difficult to understand what I found to be so satisfying about learning and using J:

• I think that partly it was the challenge - a challenge to learn a language that was very different to other languages that I had used.
• I found the clarity of documentation, at www.jsoftware.com, to be of a level that I had only previously found in old Unix, Plan 9 or Inferno documentation.
• It felt like the features of the language were necessary and sufficient to allow a particular problem to be solved.
• I can begin to see how having a terse language can help an experienced reader to visualise a complex algorithm without having to page around in source code.
• There didn’t seem to be a big overlap between the roles of the system verbs. It didn’t feel like there was supposed to be more than one way to do something (I’m reminded of Perl here).
• J has a very mathematical, and precise, feel to it.
• Using J and its ecosystem has helped to reinforce my belief that newer is certainly not always better.

Sadly, it also feels as though J might not be around for ever, and that would be a shame.

Row Totals - Counts

The mechanism can be extended to include row totals. The technique is to create a dataframe for the body of the table and another dataframe for the rowtotal and then to combine them with bind_rows.

body <- cars |>
  select(engine, new_used) |>
  group_by(engine, new_used) |>
  summarise(n = n()) |>
  pivot_wider(names_from=new_used, values_from=n, values_fill=0) 

rowtotals <- body |>
  group_by() |>
  summarise(
    across(where(is.factor), ~"Total"),
    across(where(is.numeric), sum)
  ) 
bind_rows(body, rowtotals) |>
knitr::kable(digits=0)

This is fine for simple additive counts, but for aggregates we need something better.

Row Totals - Aggregates

The problem with tables containing aggregates is that the aggregate of a column of aggregates is not the same as the aggregate you would obtain had you not previously grouped on the rows. An easy way to see this is to imagine dividing a collection of values up into two unequally sized groups. You can calculate the mean of each group but the mean of the whole collection is not equal in value to the mean of the two group means (it would be if the two groups were equally sized).

To calculate aggregates correctly for row totals we need to start with the cars dataframe and perform a different grouping for each kind of aggregate. Instead of grouping and summarising the body dataframe as we did above, we obtain row totals by grouping the cars dataframe across the table columns. This allows us to calculate an aggregate value for each group (table column) and our rowtotal is constructed as a single row dataframe that contains an aggregate value in each column and a blank value or text label in any column for which no aggregate can be calculated. We populate these blank or label columns using the mutate function.

body <- cars |>
  select(engine, new_used, dpm=depreciation_per_month) |>
  group_by(engine, new_used) |>
  summarise(dpm = median(dpm, na.rm=T)) |>
  pivot_wider(names_from=new_used, values_from=dpm, values_fill=0) 

rowtotals <- cars |>
  select(new_used, dpm=depreciation_per_month) |>
  group_by(new_used) |>
  summarise(dpm = median(dpm, na.rm=T)) |>
  pivot_wider(names_from=new_used, values_from=dpm, values_fill=0) |>
  mutate(engine = "Median")

bind_rows(body, rowtotals) |>
knitr::kable(digits=0)