Excel does provide a function for calculating new values within a pivot table. One example is extending a pivot table containing pricing and quantity data to compute an average selling price. For example, given the table below:

a new label called “ASP”, which is the Net Booking divided by the Quantity, can be added quickly and easily with Excel’s Calculated Field capability.

This feature is handy for adding labels on the fly that require a simple calculation.

In other cases, deriving the new field may not be so simple, yet needs to be performed each time the spreadsheet is updated. Python can programmatically add new data fields to the source table so that the data is ready for viewing whenever the pivot table is opened.

The script developed last time automated the data cleanup and pivot table generation tasks. Doing some further analysis based on the output spreadsheet, I created a chart of the Top 10 Customers for ABCD Catering:

Note that some of the company names are 15 characters or longer in length and occupy much of the chart space. It would be nice to have a shorter “nickname” for each company that could be used in the charts. One solution is to cut and paste the pivot table data, then modify the Company Name information by hand. Unfortunately, this would be very tedious. Another approach is to automate the process in the script and create a new column derived from a comprehensive reference table of company names and nicknames. The downside is that maintaining the list could be an issue as the business grows and the list of customers grows longer. A third method is to create an algorithm that uses the first word in the company name wherever possible, and uses a defined nickname for other special cases. “Sun Microsystems” becomes “Sun” and “Cisco Systems” becomes “Cisco”, while other company names such as “Hewlett-Packard” could be listed in a lookup with a nickname such as “HP”. The snippet below shows how this is done.

logolookup = {'Applied Materials':'AMAT', 'Electronic Arts':'EA',
              'Hewlett-Packard':'HP', 'KLA-Tencor':'KLA'}
if ("Company Name" in newdata[0]):
    cindx = newdata[0].index("Company Name")
    newdata[0][cindx+1:cindx+1] = ["Logo Name"]
    for rcnt in range(1,len(newdata)):
        if newdata[rcnt][cindx] in logolookup:
            newdata[rcnt][cindx+1:cindx+1] = [logolookup[newdata[rcnt][cindx]]]
        else:
            newname = newdata[rcnt][cindx].split()[0]
            newdata[rcnt][cindx+1:cindx+1] = [newname]
            logolookup[newdata[rcnt][cindx]] = newname

This code begins with a simple lookup for company names and can be easily extended as special case company names are added. Next, the column location of the “Company Name” field is identified and the new header “Logo Name” is inserted after “Company Name” in the list using the list[index:index] construct. The for loop iterates over each row in the table, checking whether the company name for that row exists in the logolookup dictionary, then inserting the abbreviated name. If not found, then the original company name is split() into words and the first word used as the new abbreviated name. Finally, the logolookup dictionary is updated with the new abbreviated name.

After running the program, the new column “Logo Name” has been inserted after “Company Name” and contains the shortened company names.

The new “Logo Name” column can be used in the previous pivot table and chart, replacing the “Company Name” field and producing a cleaner chart with less area used for displaying company name information.

Another use of this technique is to add a label for “Food Category” based on the type of food purchased. For example, the food items sold by ABCD Catering are: Caesar Salad, Cheese Pizza, Cheeseburger, Chocolate Sundae, Churro, Hamburger, Hot Dog, Pepperoni Pizza, Potato Chips and Soda. Let’s say that your manager wants to track the sales of different food categories, such as Burger, Dessert, HotDog, Drink, Pizza, Salad and Snack. Using the same technique outlined above, this code will add a column for Food Category with the appropriate entry for each food item:

foodlookup = {'Caesar Salad':'Salad', 'Cheese Pizza':'Pizza',
              'Cheeseburger':'Burger', 'Chocolate Sundae':'Dessert',
              'Churro':'Snack', 'Hamburger':'Burger', 'Hot Dog':'HotDog',
              'Pepperoni Pizza':'Pizza', 'Potato Chips':'Snack',
              'Soda':'Drink'}
if ("Food Name" in newdata[0]):
    cindx = newdata[0].index("Food Name")
    newdata[0][cindx+1:cindx+1] = ["Food Category"]
    for rcnt in range(1,len(newdata)):
        if newdata[rcnt][cindx] in foodlookup:
            newdata[rcnt][cindx+1:cindx+1] = [foodlookup[newdata[rcnt][cindx]]]
        else:
            newdata[rcnt][cindx+1:cindx+1] = ['UNDEFINED']

If a food item is not found in the lookup, the category is labeled UNDEFINED. This is an indication that there is a problem with the script and the lookup for food categories needs to be extended.

The section of the script which creates the pivot tables can be easily extended to build a new table based on the newly created label “Food Category”:

# What food category had the highest unit sales in Q4?
ptname = addpivot(wb,src,
         title="Unit Sales by Food Category",
         filters=("Fiscal Quarter",),
         columns=(),
         rows=("Food Category",),
         sumvalue="Sum of Quantity",
         sortfield=("Food Category",win32c.xlDescending))
wb.Sheets("Unit Sales by Food Category").PivotTables(ptname).PivotFields("Fiscal Quarter").CurrentPage = "2009-Q4"

Based on the output spreadsheet, the best selling food category in Q4 based on quantity is “Snack”, with sales of 13700 units.

Here is the completed script, also available on GitHub

#
# erppivotextended.py:
# Load raw EPR data, clean up header info,
# insert additional data fields and build 5 pivot tables
#
import win32com.client as win32
win32c = win32.constants
import sys
import itertools
tablecount = itertools.count(1)

def addpivot(wb,sourcedata,title,filters=(),columns=(),
             rows=(),sumvalue=(),sortfield=""):
    """Build a pivot table using the provided source location data
    and specified fields
    """
    newsheet = wb.Sheets.Add()
    newsheet.Cells(1,1).Value = title
    newsheet.Cells(1,1).Font.Size = 16

    # Build the Pivot Table
    tname = "PivotTable%d"%tablecount.next()

    pc = wb.PivotCaches().Add(SourceType=win32c.xlDatabase,
                                 SourceData=sourcedata)
    pt = pc.CreatePivotTable(TableDestination="%s!R4C1"%newsheet.Name,
                             TableName=tname,
                             DefaultVersion=win32c.xlPivotTableVersion10)
    wb.Sheets(newsheet.Name).Select()
    wb.Sheets(newsheet.Name).Cells(3,1).Select()
    for fieldlist,fieldc in ((filters,win32c.xlPageField),
                            (columns,win32c.xlColumnField),
                            (rows,win32c.xlRowField)):
        for i,val in enumerate(fieldlist):
            wb.ActiveSheet.PivotTables(tname).PivotFields(val).Orientation = fieldc
            wb.ActiveSheet.PivotTables(tname).PivotFields(val).Position = i+1

    wb.ActiveSheet.PivotTables(tname).AddDataField(
        wb.ActiveSheet.PivotTables(tname).PivotFields(sumvalue[7:]),
        sumvalue,
        win32c.xlSum)
    if len(sortfield) != 0:
        wb.ActiveSheet.PivotTables(tname).PivotFields(sortfield[0]).AutoSort(sortfield[1], sumvalue)
    newsheet.Name = title

    # Uncomment the next command to limit output file size, but make sure
    # to click Refresh Data on the PivotTable toolbar to update the table
    # newsheet.PivotTables(tname).SaveData = False

    return tname

def runexcel():
    """Open the spreadsheet ABCDCatering.xls, clean it up,
    and add pivot tables
    """
    excel = win32.gencache.EnsureDispatch('Excel.Application')
    #excel.Visible = True
    try:
        wb = excel.Workbooks.Open('ABCDCatering.xls')
    except:
        print "Failed to open spreadsheet ABCDCatering.xls"
        sys.exit(1)
    ws = wb.Sheets('Sheet1')
    xldata = ws.UsedRange.Value
    newdata = []
    for row in xldata:
        if len(row) == 13 and row[-1] is not None:
            newdata.append(list(row))
    lasthdr = "Col A"
    for i,field in enumerate(newdata[0]):
        if field is None:
            newdata[0][i] = lasthdr + " Name"
        else:
            lasthdr = newdata[0][i]

    logolookup = {'Applied Materials':'AMAT', 'Electronic Arts':'EA',
                  'Hewlett-Packard':'HP', 'KLA-Tencor':'KLA'}
    if ("Company Name" in newdata[0]):
        cindx = newdata[0].index("Company Name")
        newdata[0][cindx+1:cindx+1] = ["Logo Name"]
        for rcnt in range(1,len(newdata)):
            if newdata[rcnt][cindx] in logolookup:
                newdata[rcnt][cindx+1:cindx+1] = [logolookup[newdata[rcnt][cindx]]]
            else:
                newname = newdata[rcnt][cindx].split()[0]
                newdata[rcnt][cindx+1:cindx+1] = [newname]
                logolookup[newdata[rcnt][cindx]] = newname

    foodlookup = {'Caesar Salad':'Salad', 'Cheese Pizza':'Pizza',
                  'Cheeseburger':'Burger', 'Chocolate Sundae':'Dessert',
                  'Churro':'Snack', 'Hamburger':'Burger', 'Hot Dog':'HotDog',
                  'Pepperoni Pizza':'Pizza', 'Potato Chips':'Snack',
                  'Soda':'Drink'}
    if ("Food Name" in newdata[0]):
        cindx = newdata[0].index("Food Name")
        newdata[0][cindx+1:cindx+1] = ["Food Category"]
        for rcnt in range(1,len(newdata)):
            if newdata[rcnt][cindx] in foodlookup:
                newdata[rcnt][cindx+1:cindx+1] = [foodlookup[newdata[rcnt][cindx]]]
            else:
                newdata[rcnt][cindx+1:cindx+1] = ['UNDEFINED']

    rowcnt = len(newdata)
    colcnt = len(newdata[0])
    wsnew = wb.Sheets.Add()
    wsnew.Range(wsnew.Cells(1,1),wsnew.Cells(rowcnt,colcnt)).Value = newdata
    wsnew.Columns.AutoFit()

    src = "%s!R1C1:R%dC%d"%(wsnew.Name,rowcnt,colcnt)

    # What were the total sales in each of the last four quarters?
    addpivot(wb,src,
             title="Sales by Quarter",
             filters=(),
             columns=(),
             rows=("Fiscal Quarter",),
             sumvalue="Sum of Net Booking",
             sortfield=())

    # What are the sales for each food item in each quarter?
    addpivot(wb,src,
             title="Sales by Food Item",
             filters=(),
             columns=("Food Name",),
             rows=("Fiscal Quarter",),
             sumvalue="Sum of Net Booking",
             sortfield=())

    # Who were the top 10 customers for ABCD Catering in 2009?
    addpivot(wb,src,
             title="Top 10 Customers",
             filters=(),
             columns=(),
             rows=("Company Name",),
             sumvalue="Sum of Net Booking",
             sortfield=("Company Name",win32c.xlDescending))

    # Who was the highest producing sales rep for the year?
    addpivot(wb,src,
             title="Top Sales Reps",
             filters=(),
             columns=(),
             rows=("Sales Rep Name","Company Name"),
             sumvalue="Sum of Net Booking",
             sortfield=("Sales Rep Name",win32c.xlDescending))

    # What food item had the highest unit sales in Q4?
    ptname = addpivot(wb,src,
             title="Unit Sales by Food",
             filters=("Fiscal Quarter",),
             columns=(),
             rows=("Food Name",),
             sumvalue="Sum of Quantity",
             sortfield=("Food Name",win32c.xlDescending))
    wb.Sheets("Unit Sales by Food").PivotTables(ptname).PivotFields("Fiscal Quarter").CurrentPage = "2009-Q4"

    # What food category had the highest unit sales in Q4?
    ptname = addpivot(wb,src,
             title="Unit Sales by Food Category",
             filters=("Fiscal Quarter",),
             columns=(),
             rows=("Food Category",),
             sumvalue="Sum of Quantity",
             sortfield=("Food Category",win32c.xlDescending))
    wb.Sheets("Unit Sales by Food Category").PivotTables(ptname).PivotFields("Fiscal Quarter").CurrentPage = "2009-Q4"

    if int(float(excel.Version)) >= 12:
        wb.SaveAs('newABCDCatering.xlsx',win32c.xlOpenXMLWorkbook)
    else:
        wb.SaveAs('newABCDCatering.xls')
    excel.Application.Quit()

if __name__ == "__main__":
    runexcel()

Prerequisites
Python (refer to http://www.python.org)

Win32 Python module (refer to http://sourceforge.net/projects/pywin32)

Microsoft Excel (refer to http://office.microsoft.com/excel)

Source Files and Scripts
Source for the program erppivotextended.py and spreadsheet file ABCDCatering.xls are available at
http://github.com/pythonexcels/examples

Thanks — Dan

The goal of this exercise is to automate the generation of pivot tables from the last post, and save them to a new Excel file.

I started with the file newABCDCatering.xls from the previous post and record the macro to create this simple pivot table showing Net Bookings by Sales Rep and Food Name for the last four quarters.

Captured in Excel 2007, the recorded macro looks like this:

Sub Macro1()
'
' Macro1 Macro
'

'
    Selection.CurrentRegion.Select
    Sheets.Add
    ActiveWorkbook.PivotCaches.Create(SourceType:=xlDatabase, SourceData:= _
        "Sheet2!R1C1:R791C13", Version:=xlPivotTableVersion10).CreatePivotTable _
        TableDestination:="Sheet3!R3C1", TableName:="PivotTable1", DefaultVersion _
        :=xlPivotTableVersion10
    Sheets("Sheet3").Select
    Cells(3, 1).Select
    With ActiveSheet.PivotTables("PivotTable1").PivotFields("Fiscal Year")
        .Orientation = xlPageField
        .Position = 1
    End With
    With ActiveSheet.PivotTables("PivotTable1").PivotFields("Fiscal Quarter")
        .Orientation = xlColumnField
        .Position = 1
    End With
    With ActiveSheet.PivotTables("PivotTable1").PivotFields("Sales Rep Name")
        .Orientation = xlRowField
        .Position = 1
    End With
    With ActiveSheet.PivotTables("PivotTable1").PivotFields("Food Name")
        .Orientation = xlRowField
        .Position = 2
    End With
    ActiveSheet.PivotTables("PivotTable1").AddDataField ActiveSheet.PivotTables( _
        "PivotTable1").PivotFields("Net Booking"), "Sum of Net Booking", xlSum
End Sub

The post Mapping Excel VB Macros to Python covered a technique for recording a Visual Basic macro and porting it to Python. Using that approach, you could simply turn on the macro recorder and generate all the required tables, producing a long script with lots of redundancy. A better approach is to build a general purpose function that can be used over and over to generate the pivot tables.

Looking at the macro, you see lines specifying the Orientation of the field name, such as .Orientation = xlRowField and .Orientation = xlColumnField. A pivot table has four basic areas for fields:

• Report Filter (.Orientation = xlPageField)
• Column area (.Orientation = xlColumnField)
• Row area (.Orientation = xlRowField)
• Values area (PivotTables().AddDataField())

Each of these supports multiple fields (column fields for Sales Rep Name and Food Name were added in the example). The ordering of the fields changes the appearance of the table.

A general pattern should be apparent in this macro. First, the pivot table is created with the ActiveWorkbook.PivotCaches.Create() statement. Next, the columns and rows are configured with a series of ActiveSheet.PivotTables("PivotTable1").PivotFields() statements. Finally, the field used in the Values section of the table is configured using the ActiveSheet.PivotTables("PivotTable1").AddDataField statement. The general purpose function will need to contain all of these constructs. Note the parts that can’t be hard-coded: the source of the data, "Sheet2!R1C1:R791C13", and destination for the table, "Sheet3!R3C1" need to be determined based on the characteristics of the source data and can’t be hard coded in the general solution.

In Python, this pattern can be reduced to the following loop that covers fields for the Report Filter, Columns and Rows:

def addpivot(wb,sourcedata,title,filters=(),columns=(),
             rows=(),sumvalue=(),sortfield=""):
    """Build a pivot table using the provided source location data
    and specified fields
    """
    ...
    for fieldlist,fieldc in ((filters,win32c.xlPageField),
                            (columns,win32c.xlColumnField),
                            (rows,win32c.xlRowField)):
        for i,val in enumerate(fieldlist):
            wb.ActiveSheet.PivotTables(tname).PivotFields(val).Orientation = fieldc
        wb.ActiveSheet.PivotTables(tname).PivotFields(val).Position = i+1
    ...

Processing the Values field is more or less copied from the Visual Basic. To keep things simple in this example, this code is limited to adding “Sum of” values only, and doesn’t handle other Summarize Value functions such as Count, Min, Max, etc.

    wb.ActiveSheet.PivotTables(tname).AddDataField(
        wb.ActiveSheet.PivotTables(tname).PivotFields(sumvalue[7:]),
        sumvalue,
        win32c.xlSum)

The actual values for filters,columns and rows in the function are defined in the call to the function. The complete function creates a new sheet within the workbook, then adds an empty pivot table to the sheet and builds the table using the field information provided. For example, to answer the question: What were the total sales in each of the last four quarters?, the pivot table is built with the following call to the addpivot function:

# What were the total sales in each of the last four quarters?
addpivot(wb,src,
         title="Sales by Quarter",
         filters=(),
         columns=(),
         rows=("Fiscal Quarter",),
         sumvalue="Sum of Net Booking",
         sortfield=())

which defines a pivot table using the row header “Fiscal Quarter” and data value “Sum of Net Booking”. The title “Sales by Quarter” is used to name the sheet itself.

To make the output spreadsheet more understandable, the title parameter passed into the function and used as a title in each worksheet and as the tab name.

The complete script is shown below. Caveats:

• This script has been modified to run on both Excel 2007 and Excel 2003 and has been tested on those versions.
• Adding pivot tables increases the size of the output Excel file, which can be mitigated by disabling caching of pivot table data. Line 48 of the script contains the command newsheet.PivotTables(tname).SaveData = False, which has been commented out. Uncommenting this command will reduce the size of the output Excel file, but will require that the pivot table be refreshed before use by clicking on Refresh Data on the PivotTable toolbar.

#
# erpdatapivot.py:
# Load raw EPR data, clean up header info and
# build 5 pivot tables
#
import win32com.client as win32
win32c = win32.constants
import sys
import itertools
tablecount = itertools.count(1)

def addpivot(wb,sourcedata,title,filters=(),columns=(),
             rows=(),sumvalue=(),sortfield=""):
    """Build a pivot table using the provided source location data
    and specified fields
    """
    newsheet = wb.Sheets.Add()
    newsheet.Cells(1,1).Value = title
    newsheet.Cells(1,1).Font.Size = 16

    # Build the Pivot Table
    tname = "PivotTable%d"%tablecount.next()

    pc = wb.PivotCaches().Add(SourceType=win32c.xlDatabase,
                                 SourceData=sourcedata)
    pt = pc.CreatePivotTable(TableDestination="%s!R4C1"%newsheet.Name,
                             TableName=tname,
                             DefaultVersion=win32c.xlPivotTableVersion10)
    wb.Sheets(newsheet.Name).Select()
    wb.Sheets(newsheet.Name).Cells(3,1).Select()
    for fieldlist,fieldc in ((filters,win32c.xlPageField),
                            (columns,win32c.xlColumnField),
                            (rows,win32c.xlRowField)):
        for i,val in enumerate(fieldlist):
            wb.ActiveSheet.PivotTables(tname).PivotFields(val).Orientation = fieldc
            wb.ActiveSheet.PivotTables(tname).PivotFields(val).Position = i+1

    wb.ActiveSheet.PivotTables(tname).AddDataField(
        wb.ActiveSheet.PivotTables(tname).PivotFields(sumvalue[7:]),
        sumvalue,
        win32c.xlSum)
    if len(sortfield) != 0:
        wb.ActiveSheet.PivotTables(tname).PivotFields(sortfield[0]).AutoSort(sortfield[1], sumvalue)
    newsheet.Name = title

    # Uncomment the next command to limit output file size, but make sure
    # to click Refresh Data on the PivotTable toolbar to update the table
    # newsheet.PivotTables(tname).SaveData = False

    return tname

def runexcel():
    excel = win32.gencache.EnsureDispatch('Excel.Application')
    #excel.Visible = True
    try:
        wb = excel.Workbooks.Open('ABCDCatering.xls')
    except:
        print "Failed to open spreadsheet ABCDCatering.xls"
        sys.exit(1)
    ws = wb.Sheets('Sheet1')
    xldata = ws.UsedRange.Value
    newdata = []
    for row in xldata:
        if len(row) == 13 and row[-1] is not None:
            newdata.append(list(row))
    lasthdr = "Col A"
    for i,field in enumerate(newdata[0]):
        if field is None:
            newdata[0][i] = lasthdr + " Name"
        else:
            lasthdr = newdata[0][i]
    rowcnt = len(newdata)
    colcnt = len(newdata[0])
    wsnew = wb.Sheets.Add()
    wsnew.Range(wsnew.Cells(1,1),wsnew.Cells(rowcnt,colcnt)).Value = newdata
    wsnew.Columns.AutoFit()

    src = "%s!R1C1:R%dC%d"%(wsnew.Name,rowcnt,colcnt)

    # What were the total sales in each of the last four quarters?
    addpivot(wb,src,
             title="Sales by Quarter",
             filters=(),
             columns=(),
             rows=("Fiscal Quarter",),
             sumvalue="Sum of Net Booking",
             sortfield=())

    # What are the sales for each food item in each quarter?
    addpivot(wb,src,
             title="Sales by Food Item",
             filters=(),
             columns=("Food Name",),
             rows=("Fiscal Quarter",),
             sumvalue="Sum of Net Booking",
             sortfield=())

    # Who were the top 10 customers for ABCD Catering in 2009?
    addpivot(wb,src,
             title="Top 10 Customers",
             filters=(),
             columns=(),
             rows=("Company Name",),
             sumvalue="Sum of Net Booking",
             sortfield=("Company Name",win32c.xlDescending))

    # Who was the highest producing sales rep for the year?
    addpivot(wb,src,
             title="Top Sales Reps",
             filters=(),
             columns=(),
             rows=("Sales Rep Name","Company Name"),
             sumvalue="Sum of Net Booking",
             sortfield=("Sales Rep Name",win32c.xlDescending))

    # What food item had the highest unit sales in Q4?
    ptname = addpivot(wb,src,
             title="Unit Sales by Food",
             filters=("Fiscal Quarter",),
             columns=(),
             rows=("Food Name",),
             sumvalue="Sum of Quantity",
             sortfield=("Food Name",win32c.xlDescending))
    wb.Sheets("Unit Sales by Food").PivotTables(ptname).PivotFields("Fiscal Quarter").CurrentPage = "2009-Q4"

    if int(float(excel.Version)) >= 12:
        wb.SaveAs('newABCDCatering.xlsx',win32c.xlOpenXMLWorkbook)
    else:
        wb.SaveAs('newABCDCatering.xls')
    excel.Application.Quit()

if __name__ == "__main__":
    runexcel()

Prerequisites
Python (refer to http://www.python.org)

Microsoft Excel (refer to http://office.microsoft.com/excel)

Source Files and Scripts
Source for the program erpdatapivot.py and input spreadsheet file ABCDCatering.xls are available at
http://github.com/pythonexcels/examples

Thanks — Dan

In this example, a fictional company called ABCD Catering has recorded sales and order history for 2009 in their corporate ERP system. ABCD Catering provides catering services to leading Silicon Valley companies, providing the best in hamburgers, hot dogs, churros, sodas and other comfort food. Your boss has asked you to examine this data and answer some questions and produce charts representing some of the data:

• What were the total sales in each of the last four quarters?
• What are the sales for each food item in each quarter?
• Who were the top 10 customers for ABCD catering in Q1?
• Who was the highest producing sales rep for the year?
• What food item had the highest unit sales in Q4?

Generating this information typically involves running five separate reports in the system. Since your boss is looking for this same information at the end of each quarter, you want to simplify your life and your bosses by automating the report. Using Python and Excel, you can download a spreadsheet copy of the raw data, process it, generate the key figures and charts and save them to a spreadsheet.

Take a look at the data in ABCDCatering.xls:

The spreadsheet contains some header information, then a large table of records for each order. Each record contains the fiscal year and quarter, food item, company name, order data, sales representative, booking and order quantity for each order. The data needs some work before you can use it in a pivot table. First, the data in rows 1 through 11 must be ignored, it’s meaningless for the pivot table. Also, some columns do not have a proper header and must be corrected before the data can be used. The good news is that after some minor massaging, this data will be ideally suited for processing with a pivot table in Excel. Close the spreadsheet and get ready to build the reports.

The program begins with the standard boilerplate: import the win32 module and start Excel. If you have questions on this, please refer to the earlier posts.

#
# erpdata.py: Load raw EPR data and clean up header info
#
import win32com.client as win32
import sys
excel = win32.gencache.EnsureDispatch('Excel.Application')
excel.Visible = True

Next, open the spreadsheet ABCDCatering.xls with some exception handling. The try/except clause attempts to open the file with the Workbooks.Open() method, and exits gracefully if the file is missing or some other problem occurred. Lastly, the variable ws is set to the spreadsheet containing the data.

try:
    wb = excel.Workbooks.Open('ABCDCatering.xls')
except:
    print "Failed to open spreadsheet ABCDCatering.xls"
    sys.exit(1)
ws = wb.Sheets('Sheet1')

An easy way to load the entire spreadsheet into Python is the UsedRange method. The following command:

xldata = ws.UsedRange.Value

grabs all the data in the Sheet1 worksheet and copies it into a tuple named xldata. Once inside Python, the data can be manipulated and placed back into the spreadsheet with minimal calls to the COM interface, resulting in faster, more efficient processing.

To delete rows, add columns and do other operations on the data, it must be converted to or copied to a list. The approach used here is to examine the data row by row, discarding the non essential header rows and copying everything else to a new list. The first step is to remove the rows that are not part of the column header row or record data. If you are using Python to generate the program interactively, you can investigate the data in the xldata tuple and display the data for the first record (xldata[0]) and header record (xldata[11]):

The length of both rows is 13, though xldata[0] contains many elements with a value of None. The following code checks the length of the data and skips any rows shorter then 13 fields or rows that contain None in the last field. Note that this code assumes that the actual data in the table always contains complete records, true in this dataset but you should always understand the characteristics of the data you’re working on.

newdata = []
for row in xldata:
    if row[-1] is not None and len(row) == 13:
        newdata.append(row)

The newdata list now contains the header and data rows from the spreadsheet, but the header row is still not complete. All column headers must contain text in order to use this data in a pivot table. Unfortunately, the spreadsheet downloads produced by the ERP system have the column label over the numberical identifier for the item, while the text column header is blank. You can see that for the “Food” and “Company” data below.

One approach that works for this data is to scan the header and insert a column header based on the contents of the previous column. For example, the label for column F could be “Company Name”, created by simply appending the text ” Name” to the column header “Company” from the prior column. Using this simple algorithm, the column header row can be filled out and the spreadsheet made ready for pivot table conversion. A more complex lookup could be used as well, but the simple algorithm described here will scale if new fields are added to the report.

lasthdr = "Col A"
for i,field in enumerate(newdata[0]):
  if field is None:
    newdata[0][i] = lasthdr + " Name"
  else:
    lasthdr = newdata[0][i]

Now the data is ready for insertion back into the spreadsheet. To enable comparison between the new data set and the original, create a new sheet in the workbook, write the data to the new sheet and autofit the columns.

wsnew = wb.Sheets.Add()
wsnew.Range(wsnew.Cells(1,1),wsnew.Cells(len(newdata),len(newdata[0]))).Value = newdata
wsnew.Columns.AutoFit()

The last step is to save the worksheet to a new file and quit Excel. The Excel version is checked in order to save the data in the correct spreadsheet format. Version 12 corresponds to Excel 2007, which uses the .xlsx file extension. You also have to specify the constant xlOpenXMLWorkbook to define the type of output Excel file. Earlier version of Excel use the .xls extension, and because the input file was .xls format, no output format specifier is needed for users of older versions of Excel.

if int(float(excel.Version)) >= 12:
    wb.SaveAs('newABCDCatering.xlsx',win32.constants.xlOpenXMLWorkbook)
else:
    wb.SaveAs('newABCDCatering.xls')
excel.Application.Quit()

If the file newABCDCatering.xlsx or newABCDCatering.xls already exists in My Documents, you will see the following popup when you run the script.

Click “Yes” to overwrite the spreadsheet file. To run the script cleanly, erase the file newABCDCatering.xlsx or newABCDCatering.xls and try the script again.

After running the script, open the file newABCDCatering.xlsx or newABCDCatering.xls and view the contents. Note that the extraneous header information has been removed and blank column header information has been inserted programmatically as described earlier.

The new spreadsheet is ready for use in a pivot table, which will be covered in the next post. Here is the complete script, also available at github.

#
# erpdata.py: Load raw EPR data and clean up header info
#
import win32com.client as win32
import sys
excel = win32.gencache.EnsureDispatch('Excel.Application')
#excel.Visible = True
try:
    wb = excel.Workbooks.Open('ABCDCatering.xls')
except:
    print "Failed to open spreadsheet ABCDCatering.xls"
    sys.exit(1)
ws = wb.Sheets('Sheet1')
xldata = ws.UsedRange.Value
newdata = []
for row in xldata:
    if len(row) == 13 and row[-1] is not None:
        newdata.append(list(row))
lasthdr = "Col A"
for i,field in enumerate(newdata[0]):
    if field is None:
        newdata[0][i] = lasthdr + " Name"
    else:
        lasthdr = newdata[0][i]
wsnew = wb.Sheets.Add()
wsnew.Range(wsnew.Cells(1,1),wsnew.Cells(len(newdata),len(newdata[0]))).Value = newdata
wsnew.Columns.AutoFit()
if int(float(excel.Version)) >= 12:
    wb.SaveAs('newABCDCatering.xlsx',win32.constants.xlOpenXMLWorkbook)
else:
    wb.SaveAs('newABCDCatering.xls')
excel.Application.Quit()

Prerequisites
Python (refer to http://www.python.org)

Win32 Python module (refer to http://sourceforge.net/projects/pywin32)

Microsoft Excel (refer to http://office.microsoft.com/excel)

Source Files and Scripts
Source for the program erpdata.py and spreadsheet file ABCDCatering.xls are available at
http://github.com/pythonexcels/examples

Thanks — Dan

This example creates two tables from scratch — a simple multiplication table and a table of random numbers — and applies conditional formatting to the numbers using some of the new features in Excel 2007 (unfortunately this exercise won’t be compatible with older versions of Excel). Begin by starting the Python IDLE interface. Next, start Excel as you’ve done in the previous exercises. For this exercise, add a workbook using the Workbooks.Add() construct, and set the ws variable to point to the first worksheet in the workbook.

After typing these command in IDLE, you’ll see the Excel window containing an empty spreadsheet. To build the multiplication table, use Python to populate the column and row headers. There are a number of ways to do this, for this exercise you’ll pass an list of column header and row header values to Excel. A row of data is defined by using the ws.Range().Value statement with a list or tuple on right hand side of the equals sign. Rather than explicitly defining the list as [1,2,3,4,5,6,7,8,9,10], you can use a functional programming statement containing a range() statement to populate the values: [i for i in range(1,11)]. The complete statement is ws.Range("B2:K2").Value = [i for i in range(1,11)]. Defining a single column of data is a bit trickier, you must define a list of single element lists or tuples. One way to do this is to use Python’s zip() function to transpose the flat list into a list of tuples. The complete statement is ws.Range("B2:B11").Value = zip([i for i in range(1,11)]). The statements for completing the column and row headers are shown below.

At this point the column and row headers will appear in the Excel spreadsheet.

To define the product values for each cell in the table, create a formula to multiply the column and row header for a single cell, then used Excel to autofill the remaining cells. Looking at the spreadsheet, the product for cell C3 is cell B3 multiplied by cell C2, or 2 times 2 which equals 4. In terms of Excel, the formula is =B3*C2. To use Excel’s autofill capability, you need to anchor the row and column in the formula by preceding it with the $ character. In other words, the formula you want to use is =$B3*C$2. Once that formula is entered, the expansion to fill the remaining cells is done in two steps. First, programmatically select the cell and drag it fill the row. Next, select the newly autofilled row and drag the new row down to fill in the remaining rows. Since this was demonstrated in the last post, please refer to that post if you need more information. The equivalent Python code to implement the autofill is shown below.

The spreadsheet will now contain the complete multiplication table.

To help illustrate conditional formatting, create another table of random integers between 1 and 100. Excel’s RAND() function will generate a random number between 0 and 1, the formula we want is =INT(RAND()*100). The ws.Range().Formula construct can be used to fill a range with the same identical formula.

The Excel spreadsheet should now contain both the multiplication and random number tables.

Now that the data is ready, conditional formatting can be applied. Even though you invoked Excel from Python, you still can manipulate the spreadsheet using the Excel interface, and even record macros. You need to record a macro like you did in the last post in order to capture the VB commands, so click on Record Macro in the Developer tab, then click OK in the popup dialog.

Select all the cells in the range B2:K22. In the Home tab, select Conditional Formatting->Color Scales->Red-Yellow-Blue Color Scale.

The spreadsheet should now show a color background for each of the selected cells containing a value. Now select cell A1, then stop the macro by clicking Stop Recording in the Developer tab.

Your spreadsheet will now have conditional formatting applied and will look something like this, with cells containing numbers near 100 colored in Red, cells with a value of 50 in Yellow, and cells with a value of 1 in Blue, with a shade of these colors for values in between:

As an aside, you can update the random numbers in the lower table by hitting the F9 key to force a spreadsheet recalculation.

To continue, open the macro just created by selecting Macros from the Developer tab, select the name of the macro you just captured and click Edit. The macro should look something like this:

Sub Macro1()
'
' Macro1 Macro
'
    Range("B2:K22").Select
    Selection.FormatConditions.AddColorScale ColorScaleType:=3
    Selection.FormatConditions(Selection.FormatConditions.Count).SetFirstPriority
    Selection.FormatConditions(1).ColorScaleCriteria(1).Type = _
        xlConditionValueLowestValue
    With Selection.FormatConditions(1).ColorScaleCriteria(1).FormatColor
        .Color = 13011546
        .TintAndShade = 0
    End With
    Selection.FormatConditions(1).ColorScaleCriteria(2).Type = _
        xlConditionValuePercentile
    Selection.FormatConditions(1).ColorScaleCriteria(2).Value = 50
    With Selection.FormatConditions(1).ColorScaleCriteria(2).FormatColor
        .Color = 8711167
        .TintAndShade = 0
    End With
    Selection.FormatConditions(1).ColorScaleCriteria(3).Type = _
        xlConditionValueHighestValue
    With Selection.FormatConditions(1).ColorScaleCriteria(3).FormatColor
        .Color = 7039480
        .TintAndShade = 0
    End With
    Range("A1").Select
End Sub

Though the macro contains some very long method names, plus some With statements, the porting will be very straightforward. Here are some guidelines to keep in mind while migrating this code to Python.

Selection is preceded by excel.

Remember that Selection is a method at the Excel Application level, you need to precede it with excel. in this example.

Range is preceded by ws.
Range is a method at the Worksheet level, which is defined earlier as ws. in this example.

Function calls require () in Python
Unlike VB, any function calls must by followed by () in Python.

With statements must be expanded
The three With blocks in this macro need to be expanded, which can be done with temporary variables or by copying the statement following the With keyword. For example, the first With block:

With Selection.FormatConditions(1).ColorScaleCriteria(1).FormatColor
    .Color = 13011546
    .TintAndShade = 0
End With

can be written in Python as

excel.Selection.FormatConditions(1).ColorScaleCriteria(1).FormatColor.Color = 13011546
excel.Selection.FormatConditions(1).ColorScaleCriteria(1).FormatColor.TintAndShade = 0

or by using a temporary variable as

x = excel.Selection.FormatConditions(1).ColorScaleCriteria(1).FormatColor
x.Color = 13011546
x.FormatColor.TintAndShade = 0

Temporary variables were created to make the script more concise. In particular, the statement [csc1,csc2,csc3] = [excel.Selection.FormatConditions(1).ColorScaleCriteria(n) for n in range(1,4)] was used to create three temporary variables for the three ColorScaleCriteria methods. The equivalent Python text representing the macro is shown here:

To save the spreadsheet and close Excel, use the SaveAs and Quit methods as shown below.

Here is the complete conditionalformatting.py script. The line excel.Visible = True has been commented out. Unless you are developing the script, you typically want Excel to run invisibly in the background.

#
# conditionalformatting.py
# Create two tables and apply Conditional Formatting
#
import win32com.client as win32
excel = win32.gencache.EnsureDispatch('Excel.Application')
#excel.Visible = True
wb = excel.Workbooks.Add()
ws = wb.Worksheets('Sheet1')
ws.Range("B2:K2").Value = [i for i in range(1,11)]
ws.Range("B2:B11").Value = zip([i for i in range(1,11)])
ws.Range("C3").Formula = "=$B3*C$2"
ws.Range("C3:C3").Select()
excel.Selection.AutoFill(ws.Range("C3:K3"),win32.constants.xlFillDefault)
ws.Range("C3:K3").Select()
excel.Selection.AutoFill(ws.Range("C3:K11"),win32.constants.xlFillDefault)
ws.Range("B13:K22").Formula = "=INT(RAND()*100)"
ws.Range("B2:K22").Select()
excel.Selection.FormatConditions.AddColorScale(ColorScaleType = 3)
excel.Selection.FormatConditions(excel.Selection.FormatConditions.Count).SetFirstPriority()
[csc1,csc2,csc3] = [excel.Selection.FormatConditions(1).ColorScaleCriteria(n) for n in range(1,4)]
csc1.Type = win32.constants.xlConditionValueLowestValue
csc1.FormatColor.Color = 13011546
csc1.FormatColor.TintAndShade = 0
csc2.Type = win32.constants.xlConditionValuePercentile
csc2.Value = 50
csc2.FormatColor.Color = 8711167
csc2.FormatColor.TintAndShade = 0
csc3.Type = win32.constants.xlConditionValueHighestValue
csc3.FormatColor.Color = 7039480
csc3.FormatColor.TintAndShade = 0
ws.Range("A1").Select()
wb.SaveAs('ConditionalFormatting.xlsx')
excel.Application.Quit()

Prerequisites
Python (refer to http://www.python.org)

Win32 Python module (refer to http://sourceforge.net/projects/pywin32)

Microsoft Excel (refer to http://office.microsoft.com/excel)

Source Files and Scripts
Source for the program conditionalformatting.py script is available at
http://github.com/pythonexcels/examples

That’s all for now, thanks — Dan

This exercise will illustrate how to capture a simple set of operations in a macro, examine the Visual Basic macro, port it to Python and run it. I’m using the MultiplicationTable.xlsx file as a starting point, it’s a simple 10×10 multiplication table that will be expanded and reformatted. The first step is to capture the macro in Excel using Record Macro. In Excel 2007 the Developer tab containing the Record Macro button is turned off by default, you will need to enable it by selecting “Excel Options” from the ribbon menu, then select “Popular” in the left hand column and select the “Show Developer tab in the Ribbon” checkbox as shown here.

Starting with a simple spreadsheet containing a table of data, click on the “Developer” tab, then “Record Macro”.

If you’re using an older version of Excel, select Tools->Macro->Record New Macro from the menu as shown here.

The goal is to expand the existing table to a 15×15 table, adjust the column width to make the table appear more square and save the new spreadsheet. Now that the macro is recording, the first step is to select the last row of data and expanding it by dragging it down an additional 5 rows. First the data is selected

then dragged to create 5 new rows of data.

Do the same select and drag operation for the last column of data to create 5 new columns.

Now you have a 15×15 multiplication table. To resize the columns, select the headers for columns B through P, click the right mouse and select “Column Width”.

Enter “4″ as the new column width and click OK. The spreadsheet will now look like this:

Now stop capturing the macro by clicking on Stop Recording

If you’re using an older version of Excel, select Tools->Macro->Stop Recording from the menu bar.

To view the macro, click on the View Macros button

For older versions of Excel, select Tools->Macro->Macros

Select the macro you just recorded (this should be Macro1, but if you were experimenting you may have other macros, so select the highest numbered macro) and click Edit.

This will open your macro in the Microsoft Visual Basic GUI, and it should look something like this

Sub Macro1()
'
' Macro1 Macro
'

'
    Range("B11:K11").Select
    Selection.AutoFill Destination:=Range("B11:K16"), Type:=xlFillDefault
    Range("B11:K16").Select
    Range("K2:K16").Select
    Selection.AutoFill Destination:=Range("K2:P16"), Type:=xlFillDefault
    Range("K2:P16").Select
    Columns("B:P").Select
    Selection.ColumnWidth = 4
End Sub

Don’t worry if there are some extra or redundant lines in your macro, they can be removed as the script is ported. Now we’re ready to fire up Python and integrate this macro into a script.

Porting
To get started, start the Python Integrated Development Enviroment (IDLE), then open the spreadsheet with the 10×10 multiplication table by entering the following four commands (make sure the file “MultiplicationTable.xlsx” is in your My Documents folder.

Your screen should now look like this:

These are boilerplate commands you’ll be using in each exercise. The first two commands, import win32com.client as win32, and excel = win32.gencache.EnsureDispatch( 'Excel.Application'), import the win32 module and open the Excel process. The command wb = excel.Workbooks.Open('MultiplicationTable.xlsx') opens the worksheet. In general, you’ll need a excel.Workbooks.Open() or excel.Workbooks.Add() command to open an existing workbook or create a new workbook. The command excel.Visible = True makes Excel visible on the screen, rather than running as a hidden process in the background.

Looking at the Macro1 macro, the first command is Range("B11:K11").Select. The Range variable name is within the context of the Worksheet, so you need to create a container for operations on the worksheet. The command ws = wb.Worksheets('Sheet1') will do the trick.

If you noticed, I made a typo when entering the command and typed Worksheet instead of Worksheets. Don’t panic if you make a mistake as I did, in most cases you can simply retype the correct command and continue on.

Once the variable pointing to the worksheet is defined, append the macro command to ws. and try it. Note that Select is a function and requires the open and close parenthesis pair in order to operate correctly. This pattern may be used for every Range().Select line in the macro.

If you bring the worksheet to the foreground, you’ll see that the range B11:K11 has been selected. The next task is to autofill the 5 rows below using the Selection.AutoFill Destination:=Range("B11:K16"), Type:=xlFillDefault construct. Selection is a method at the Excel Application level, you need to precede it with excel. in this example. The arguments Destination:=Range("B11:K16"), Type:=xlFillDefault must be provided to the function, either using the keyword arguments Destination and Type, or by using positional notation. To make your programs as robust as possible, you should include the keywords, but it’s not strictly required and I don’t use that pattern in this example.

The definition for the constant xlFillDefault is contained in win32.constants, you can access this value by specifying win32.constants.xlFillDefault. I’ve seen many examples where the developer replaces this with the actual value (0 in this case). My preference is to avoid replacing Excel variables with numbers in my scripts, I believe that including the variable names increases the clarity of the script. My preference is to use the fully specified name wherever possible, but if you have to replace the variable with the actual value, you can always use the Object Browser in the VB window to figure out the correct value (open the Object Browser by pressing F2, or by selecting View->Object Browser from the menu in the VB window.

Combining these translations, the full Python command is excel.Selection.AutoFill( Destination=ws.Range("B11:K16"), Type=win32.constants.xlFillDefault ), or excel.Selection.AutoFill( ws.Range("B11:K16"), win32.constants.xlFillDefault) as I’ve used in the example.

Occasionally you’ll make a mistake when capturing a macro and record extraneous, unnecessary commands. The command Range("B11:K16").Select isn’t needed and can be ignored. The next two macro commands, Range("K2:K16").Select and Selection.AutoFill Destination:=Range("K2:P16"), Type:=xlFillDefault, are translated in the same way as the Select and AutoFill commands discussed earlier.

The commands Range("K2:K16").Select and Selection.AutoFill Destination:=Range("K2:P16"), Type:=xlFillDefault are translated in the same fashion as the earlier Select and AutoFill commands as shown below.

The worksheet is now expanded to the full 15×15 table and looks like this:

The next section of the macro selects columns B through P and sets their width to 4. The statement Columns("B:P").Select is a property of the worksheet, so prefix it with the ws. identifier and add the parenthesis to make it a Python function call. In the next statement, Selection is a property of excel, so prefix it as such. The translated statements are shown below.

The Excel spreadsheet is now complete, the multiplication table has been expanded to 15×15 and the columns have been resized to 4. At this point, translation of the macro is complete, but the modified file has to be saved. To write the file and quit Excel, use the SaveAs and Quit methods as shown below.

For your reference, here is the completed Python script.

#
# make15x15.py
# Expand an existing 10x10 multiplication table and resize columns
#
import win32com.client as win32
excel = win32.gencache.EnsureDispatch('Excel.Application')
wb = excel.Workbooks.Open('MultiplicationTable.xlsx')
excel.Visible = True
ws = wb.Worksheets('Sheet1')
ws.Range("B11:K11").Select()
excel.Selection.AutoFill(ws.Range("B11:K16"),win32.constants.xlFillDefault)
ws.Range("K2:K16").Select()
excel.Selection.AutoFill(ws.Range("K2:P16"),win32.constants.xlFillDefault)
ws.Columns("B:P").Select()
excel.Selection.ColumnWidth = 4
wb.SaveAs('NewMultiplicationTable.xlsx')
excel.Application.Quit()

If this is the first time you’ve ported an Excel macro from VB to Python, congratulations! Please note that in this example, things are kept simple and there is absolutely no error checking illustrated. Normally you would need to provide at least a minimal level of error checking in your script so that common errors (missing input file, can’t invoke Excel, etc) are caught and handled nicely. Also, this example was developed using Excel 2007, but you can run this code verbatim on earlier versions of Excel if you change the .xlsx file extension to .xls throughout the script.

Some Porting Guidelines

1. Prefix the Range().Select statements with the variable name pointing to the worksheet (ws in this example)
2. Append () to any functions
3. Prefix the Selection statements with the variable name for the Excel spreadsheet (excel in this example)
4. Prefix the Columns statements with the variable name for the worksheet(ws in this example)

Porting Reference Table for this example
Note that I didn’t capture the Workbooks.Open() or Workbooks.SaveAs lines in the VB script, it’s left as an exercise for the reader to research those commands.

Prerequisites
Python (refer to http://www.python.org)

Win32 Python module (refer to http://sourceforge.net/projects/pywin32)

Microsoft Excel (refer to http://office.microsoft.com/excel)

Source Files and Scripts
Source for the program make15×15.py and data text file are available at
http://github.com/pythonexcels/examples

That’s all for now, thanks — Dan

1. These examples were tested in Excel 2007, they should work fine in earlier versions as well after changing the extension of the file within the wb.SaveAs() statement from .xlsx to .xls
2. If you’re new to this, I recommend typing these examples by hand into IDLE, IPython or the Python interpreter, then watching the effect in Excel as you enter the commands. To make Excel visible add the line

    

   excel.Visible = True

    

   after the excel = win32.gencache.EnsureDispatch('Excel.Application') line in the script

3. These are simple examples with no error checking. Make sure the output files doesn’t exist before running the script. If the script crashes, it may leave a copy of Excel running in the background. Open the Windows Task Manager and kill the background Excel process to recover.
4. These examples contain no optimization. You typically wouldn’t use a for loop to iterate through data in individual cells, it’s provided here for illustration only.

Open Excel, Add a Workbook
The following script simply invokes Excel, adds a workbook and saves the empty workbook.

#
# add_a_workbook.py
# Add a workbook and save (Excel 2007)
# For older versions of excel, use the .xls file extension
#
import win32com.client as win32
excel = win32.gencache.EnsureDispatch('Excel.Application')
wb = excel.Workbooks.Add()
wb.SaveAs('add_a_workbook.xlsx')
excel.Application.Quit()

Open an Existing Workbook
This script opens an existing workbook and displays it (note the statement excel.Visible = True). The file workbook1.xlsx would exist in the user’s “My Documents” directory. You can also open spreadsheet files by specifying the full path to the file as shown below. Using r' in the statement r'C:\myfiles\excel\workbook2.xlsx' automatically escapes the backslash characters and makes the file name a bit more concise.

#
# open_an_existing_workbook.py
# Open an existing workbook
#
import win32com.client as win32
excel = win32.gencache.EnsureDispatch('Excel.Application')
wb = excel.Workbooks.Open('workbook1.xlsx')
# Alternately, specify the full path to the workbook
# wb = excel.Workbooks.Open(r'C:\myfiles\excel\workbook2.xlsx')
excel.Visible = True

Add a Worksheet
This script creates a new workbook with three sheets, adds a fourth worksheet and names it MyNewSheet.

#
# add_a_worksheet.py
# Add a workbook, add a worksheet,
# name it 'MyNewSheet' and save
#
import win32com.client as win32
excel = win32.gencache.EnsureDispatch('Excel.Application')
wb = excel.Workbooks.Add()
ws = wb.Worksheets.Add()
ws.Name = "MyNewSheet"
wb.SaveAs('add_a_worksheet.xlsx')
excel.Application.Quit()

Ranges and Offsets
This script uses some different techniques for addressing cells using the Cells() and Range() operators. Individual cells can be addressed using Cells(row,column), where row is the row number, column is the column number, both start from 1. Groups of cells can be addressed using Range(), where the argument in the parenthesis can be a single cell denoted by its textual name (eg "A2"), a group noted by a textual name with a colon (eg "A3:B4") or a group denoted with two Cells() identifiers (eg ws.Cells(1,1),ws.Cells(2,2)). The Offset method provides a way to address a cell based on a reference to another cell.

#
# ranges_and_offsets.py
# Using ranges and offsets
#
import win32com.client as win32
excel = win32.gencache.EnsureDispatch('Excel.Application')
wb = excel.Workbooks.Add()
ws = wb.Worksheets("Sheet1")
ws.Cells(1,1).Value = "Cell A1"
ws.Cells(1,1).Offset(2,4).Value = "Cell D2"
ws.Range("A2").Value = "Cell A2"
ws.Range("A3:B4").Value = "A3:B4"
ws.Range("A6:B7,A9:B10").Value = "A6:B7,A9:B10"
wb.SaveAs('ranges_and_offsets.xlsx')
excel.Application.Quit()

Autofill Cell Contents
This script uses Excel’s autofill capability to examine data in cells A1 and A2, then autofill the remaining column of cells through A10.

#
# autofill_cells.py
# Autofill cell contents
#
import win32com.client as win32
excel = win32.gencache.EnsureDispatch('Excel.Application')
wb = excel.Workbooks.Add()
ws = wb.Worksheets("Sheet1")
ws.Range("A1").Value = 1
ws.Range("A2").Value = 2
ws.Range("A1:A2").AutoFill(ws.Range("A1:A10"),win32.constants.xlFillDefault)
wb.SaveAs('autofill_cells.xlsx')
excel.Application.Quit()

Cell Color
This script illustrates adding an interior color to the cell using Interior.ColorIndex. Column A, rows 1 through 20 are filled with a number and assigned that ColorIndex.

#
# cell_color.py
# Add an interior color to cells
#
import win32com.client as win32
excel = win32.gencache.EnsureDispatch('Excel.Application')
wb = excel.Workbooks.Add()
ws = wb.Worksheets("Sheet1")
for i in range (1,21):
    ws.Cells(i,1).Value = i
    ws.Cells(i,1).Interior.ColorIndex = i
wb.SaveAs('cell_color.xlsx')
excel.Application.Quit()

Column Formatting
This script creates two columns of data, one narrow and one wide, then formats the column width with the ColumnWidth property. You can also use the Columns.AutoFit() function to autofit all columns in the spreadsheet.

#
# column_widths.py
# Set column widths
#
import win32com.client as win32
excel = win32.gencache.EnsureDispatch('Excel.Application')
wb = excel.Workbooks.Add()
ws = wb.Worksheets("Sheet1")
ws.Range("A1:A10").Value = "A"
ws.Range("B1:B10").Value = "This is a very long line of text"
ws.Columns(1).ColumnWidth = 1
ws.Range("B:B").ColumnWidth = 27
# Alternately, you can autofit all columns in the worksheet
# ws.Columns.AutoFit()
wb.SaveAs('column_widths.xlsx')
excel.Application.Quit()

Copying Data from Worksheet to Worksheet
This script uses the FillAcrossSheets() method to copy data from one location to all other worksheets in the workbook. Specifically, the data in the range A1:J10 is copied from Sheet1 to sheets Sheet2 and Sheet3.

#
# copy_worksheet_to_worksheet.py
# Copy data and formatting from a range of one worksheet
# to all other worksheets in a workbook
#
import win32com.client as win32
excel = win32.gencache.EnsureDispatch('Excel.Application')
wb = excel.Workbooks.Add()
ws = wb.Worksheets("Sheet1")
ws.Range("A1:J10").Formula = "=row()*column()"
wb.Worksheets.FillAcrossSheets(wb.Worksheets("Sheet1").Range("A1:J10"))
wb.SaveAs('copy_worksheet_to_worksheet.xlsx')
excel.Application.Quit()

Format Worksheet Cells
This example creates two columns of data, then formats the font type and font size used in the worksheet. Five different fonts and sizes are used, the numbers are formatted using a monetary format.

#
# format_cells.py
# Format cell font name and size, format numbers in monetary format
#
import win32com.client as win32
excel = win32.gencache.EnsureDispatch('Excel.Application')
wb = excel.Workbooks.Add()
ws = wb.Worksheets("Sheet1")

for i,font in enumerate(["Arial","Courier New","Garamond","Georgia","Verdana"]):
    ws.Range(ws.Cells(i+1,1),ws.Cells(i+1,2)).Value = [font,i+i]
    ws.Range(ws.Cells(i+1,1),ws.Cells(i+1,2)).Font.Name = font
    ws.Range(ws.Cells(i+1,1),ws.Cells(i+1,2)).Font.Size = 12+i

ws.Range("A1:A5").HorizontalAlignment = win32.constants.xlRight
ws.Range("B1:B5").NumberFormat = "$###,##0.00"
ws.Columns.AutoFit()
wb.SaveAs('format_cells.xlsx')
excel.Application.Quit()

Setting Row Height
Similar to column height, row height can be set with the RowHeight method. You can also use AutoFit() to automatically adjust the row height based on cell contents.

#
# row_height.py
# Set row heights and align text within the cell
#
import win32com.client as win32
excel = win32.gencache.EnsureDispatch('Excel.Application')
wb = excel.Workbooks.Add()
ws = wb.Worksheets("Sheet1")
ws.Range("A1:A2").Value = "1 line"
ws.Range("B1:B2").Value = "Two\nlines"
ws.Range("C1:C2").Value = "Three\nlines\nhere"
ws.Range("D1:D2").Value = "This\nis\nfour\nlines"
ws.Rows(1).RowHeight = 60
ws.Range("2:2").RowHeight = 120
ws.Rows(1).VerticalAlignment = win32.constants.xlCenter
ws.Range("2:2").VerticalAlignment = win32.constants.xlCenter

# Alternately, you can autofit all rows in the worksheet
# ws.Rows.AutoFit()

wb.SaveAs('row_height.xlsx')
excel.Application.Quit()

Prerequisites
Python (refer to http://www.python.org)

Win32 Python module (refer to http://sourceforge.net/projects/pywin32)

Microsoft Excel (refer to http://office.microsoft.com/excel)

Source Files and Scripts
Source for the program and data text file are available at
http://github.com/pythonexcels/examples

That’s all for now, thanks — Dan

For starters, let’s do some simple operations using Python to invoke Excel, add a spreadsheet, insert some data, then save the results to a spreadsheet file. You can play along at home by following my lead and entering the program text exactly as I’ve described below. My exercises and screen shots are done with Excel 2007, but all the commands presented here also work fine for Excel 2003. A prerequisite for this exercise is Python, the Win32 module and a copy of Microsoft Excel.

Here is the complete script we’ll be entering using IDLE, the Python interactive development tool.

#
# driving.py
#
import win32com.client as win32
excel = win32.gencache.EnsureDispatch('Excel.Application')
excel.Visible = True
wb = excel.Workbooks.Add()
ws = wb.Worksheets('Sheet1')
ws.Name = 'Built with Python'
ws.Cells(1,1).Value = 'Hello Excel'
print ws.Cells(1,1).Value
for i in range(1,5):
    ws.Cells(2,i).Value = i  # Slow!
ws.Range(ws.Cells(3,1),ws.Cells(3,4)).Value = [5,6,7,8]
ws.Range("A4:D4").Value = [i for i in range(9,13)]
ws.Cells(5,4).Formula = '=SUM(A2:D4)'
ws.Cells(5,4).Font.Size = 16
ws.Cells(5,4).Font.Bold = True

What follows is a step-by-step guide to entering this script and monitoring the result.

1) Open the Python IDLE interface from the Start menu

“IDLE is the Python IDE built with the tkinter GUI toolkit”, as described in the Python IDLE documentation, and gives you an interactive interface to enter, run and save Python programs. IDLE isn’t strictly necessary for this exercise, you could use any shell command window, or a tool such as IPython or the MS Windows command line interface.

2) Import the win32 module

If the import command was successful, you’ll see the “>>>” prompt returned. If there was a problem, such as not having the win32 module installed correctly, you’ll see Import Error: No module named win32com.client. In that case, install the appropriate win32 module from the web site.

3) Start Excel

The command win32.gencache.EnsureDispatch('Excel.Application') will attach to an Excel process that is already running, or will start Excel if it’s not. If you see the “>>>” prompt, Excel has been started or linked successfully. At this point you won’t see Excel, but if you check your task manager you can confirm that the process is running.

4) Make Excel Visible

Setting the Visible flag with excel.Visible = True makes the Excel window appear. At this point, Excel does not contain any workbooks or worksheets, we’ll add those in the next step.

5) Add a workbook, select the sheet “Sheet1″ and rename it

Excel needs a workbook to serve as a container for the worksheets. A new workbook containing 3 sheets is added with command wb = excel.Workbooks.Add(). The command ws = wb.Worksheets('Sheet1') assigns ws to the sheet named Sheet1, and the command ws.Name = 'Built with Python' changes the name of Sheet1. Your screen should now look something like this:

6) Add some text into the first cell

Now the setup is complete and you can add data to the spreadsheet. There are several options for addressing cells and blocks of data in Excel, we’ll cover a few of them here. You can address individual cells with the Cells(row,column).Value pattern, where row and column are integer values representing the row and column location for the cell. Use .Value to add text, numbers and date information to the cell, use .Formula for entering an Excel formula into the cell location.

After these commands, you’ll see the “Hello Excel” text in your Excel worksheet, and see the text printed in the IDLE window as well. Of course, Python can set values in the spreadsheet as well as query data from the speadsheet.

7) Populate the second row with data

In many cases you’ll have lists of data to insert into or extract from the worksheet. Wrapping the Cells(row,column).Value pattern with a loop seems like a natural approach, but in reality this maximizes the communication overhead between Python and Excel and results in very inefficient and slow code. It’s much better to transfer lists than individual elements whenever possible as shown in the next section. After this command, your Excel spreadsheet will look like this:

8) Populate the third and fourth rows of data

A better approach to populating or extracting blocks of data is to use the Range().Value pattern. With this construct you can efficiently transfer a one- or two-dimensional blocks of data. In the first example, cells (3,1) through (3,4) are assigned to the list [5,6,7,8]. The next line uses the Excel-style cell address “A4:D4″ to assign the results of the operation [i for i in range(9,13)]. In some cases, it may be more intuitive to use the Excel-style naming. The Excel sheet now looks like this:

9) Assign a formula to sum the numbers just added

You can insert Excel formulas into cells using the .Formula pattern. The formula is the same as if you were to enter it in Excel: =SUM(A2:D4). In this example, the sum of 12 numbers in rows 2,3 and 4 is generated. Your Excel sheet should now look like the screenshot below.

10) Change the formatting of the formula cell

As a final exercise, the format of the formula cell is changed to point size 16 with a bold typeface. You can change any of dozens of attributes for the various cells in the worksheet through Python. Your spreadsheet should now look like this.

Hopefully you did this exercise interactively, typing the commands and monitoring the result in Excel. You can also cut to the chase and run this script to generate the result. When the script exits, you’ll be left with an open Excel spreadsheet just as shown in the last screenshot above.

Prerequisites
Python (refer to http://www.python.org)

Win32 Python module (refer to http://sourceforge.net/projects/pywin32)

Microsoft Excel

Source Files and Scripts
Source for the program and data text file are available at
http://github.com/pythonexcels/examples/tree/master

References

Thanks everyone — Dan

• Downloading data directly from the web
• Parsing the data, removing extraneous information at the top of the file and writing the real data fields to a spreadsheet
• Adding a hyperlink for each record that links to a Google Map, based on the latitude and longitude data given

Note that it’s possible to do these steps using a web browser and an interactive Excel session. You can easily download the file, import it into Excel, remove the leading text, and make a formula to produce a hyperlink. But the beauty of this script is that everything is done automatically, which can be very handy if the source data is constantly updated.

Here’s the script that performs these operations:

#
# xlwt_bostonhousing.py
#
import sys
from urllib2 import urlopen
from xlwt import Workbook, easyxf, Formula

def doxl():
    '''Read the boston_corrected.txt file based on
       Harrison, David, and Daniel L. Rubinfeld, "Hedonic Housing Prices
       and the Demand for Clean Air," Journal of Environmental Economics
       and Management, Volume 5, (1978), write to an excel spreadsheet .
       '''
    URL = 'http://stat.cmu.edu/datasets/boston_corrected.txt'
    try:
        fp = urlopen(URL)
    except:
        print 'Failed to download %s' % URL
        sys.exit(1)
    lines = fp.readlines()

    wb = Workbook()
    ws = wb.add_sheet('Housing Data')
    ulstyle = easyxf('font: underline single')
    r = 0
    for line in lines:
        tokens = line.strip().split('\t')
        if len(tokens) != 21:
            continue
        for c,t in enumerate(tokens):
            for dtype in (int,float):
                try:
                    t = dtype(t)
                except:
                    pass
                else:
                    break
            ws.write(r,c+1,t)
        if r == 0:
            hdr = tokens
            ws.write(r,0,'MAPLINK')
        else:
            d = dict(zip(hdr,tokens))
            link = 'HYPERLINK("http://maps.google.com/maps?q=%s,+%s+(Observation+%s)&hl=en&ie=UTF8&z=14&iwloc=A";"MAP")' % (d['LAT'],d['LON'],d['OBS.'])
            ws.write(r,0,Formula(link),ulstyle)

        r += 1
    wb.save('bostonhousing.xls')
    print 'Wrote bostonhousing.xls'

if __name__ == "__main__":
    doxl()

As in the previous post, you must have xlwt installed (please refer to the site http://www.python-excel.org for information on downloading and installing). Looking at the important bits in the script above, the lines

    URL = 'http://stat.cmu.edu/datasets/boston_corrected.txt'
    try:
        fp = urlopen(URL)
    except:
        print 'Failed to download %s' % URL
        sys.exit(1)
    lines = fp.readlines()

open the URL for the boston_corrected.txt file, then reads the URL and returns a list of strings. The next section:

    wb = Workbook()
    ws = wb.add_sheet('Housing Data')
    ulstyle = easyxf('font: underline single')
    r = 0

creates a new Workbook object, then adds a sheet named “Housing Data” to the workbook. The easyfx function provides a convenient way to add formatting to the spreadsheet; in this example, the single underline format is used to denote a hyperlink. In the next line, the variable r acts as a row counter.

The for loop below examines each row of data:

    for line in lines:
        tokens = line.strip().split('\t')
        if len(tokens) != 21:
            continue
        for c,t in enumerate(tokens):
            for dtype in (int,float):
                try:
                    t = dtype(t)
                except:
                    pass
                else:
                    break
            ws.write(r,c+1,t)

Each line is “stripped” (leading and trailing white space characters are removed), then split by tab characters. A data line contains 21 fields of information, otherwise it is rejected. To properly format the data for the spreadsheet, the datatype is set using try-except-else within the for loop. The loop only considers string, integer and float data, which is sufficient for this input data. More complex input files may contain date information which would require additional handling. The cell data with the correct type setting is written to the spreadsheet using the ws.write statement.

The next section builds the hyperlink to a Google Map using the latitude and longitude information within the input data.

        if r == 0:
            hdr = tokens
            ws.write(r,0,'MAPLINK')
        else:
            d = dict(zip(hdr,tokens))
            link = 'HYPERLINK("http://maps.google.com/maps?q=%s,+%s+(Observation+%s)&hl=en&ie=UTF8&z=14&iwloc=A";"MAP")' % (d['LAT'],d['LON'],d['OBS.'])
            ws.write(r,0,Formula(link),ulstyle)
        r += 1
    wb.save('bostonhousing.xls')
    print 'Wrote bostonhousing.xls'

If this is the first row of data (if r == 0), it is assumed to be header data and is saved in the hdr variable. Otherwise, the statement d = dict(zip(hdr,tokens)) builds a dictionary, using the header information as keys. This allows each field to be referenced by its column heading. The hyperlink is built by specifying a URL containing http://maps.google.com/maps?, with the corresponding latitude (d['LAT']) and longitude (d['LON']) information from the current line of data. Finally, the hyperlink data is written to the spreadsheet with ws.write. The last two lines write the spreadsheet and print a message.

The original raw data looks like this:

The bostonhousing.xls spreadsheet containing the original data and the new map hyperlink looks like this:

Clicking on the MAP link brings up a Google Map showing the location according to the latitude and longitude. For some reason, the location of latitude 42.255000 and longitude -70.955000 isn’t in the town of Nahant, but in the middle of Rock Island Cove near Quincy. It’s left as an exercise for the reader to determine why this is so.

xlwt (refer to http://www.python-excel.org)

Source Files and Scripts
Source for the program and data text file are available at
http://github.com/pythonexcels/xlwt/tree/master

References

Thanks everyone — Dan

I’ve written a fair number of Python scripts that interface with Excel through the COM interface, but for many tasks you don’t need to invoke Excel to operate on spreadsheet data. Although some advanced operations like charting and pivot tables are not currently supported, these utilities provide an excellent way of interfacing with Excel spreadsheets for many common jobs.

In my example below, I format a raw text file containing information on reference account categories for a hospital (based on data from data.gov):

09700RESEARCH
09800PHYSICIANS  PRIVATE OFFICES
09900NONPAID WORKERS
MANAGEMENT FEES
REFERENCE LABS

The raw data is smashed together, with the account number and name running into each other without separation. Some of the data doesn’t even have an account number at all. The script accomplishes the following:

• Separate the account number and name, and write them into two separate columns in the spreadsheet
• Assign an account number of 99999 for unnumbered accounts, and set the cell color to red
• Convert the upper cased account name to a properly capitalized name, remove any extra spaces
• Format the column width of each spreadsheet column based on the width of the data

Based on the input above, the goal is to produce Excel output like this:

Excel output sample

Since there are no charts, pivot tables or other fancy formatting needed, xlwt is perfect for the job. Here is the script that performs the conversion:

#
#
import sys
import re
from xlwt import Workbook, easyxf

def doxl():
    '''Read raw account number and name strings, separate the data and
       write to an excel spreadsheet.  Properly capitalize the account
       names and mark cells with no account number as 99999 with red fill
       '''
    try:
        fp = open("hospdata.txt")
    except:
        print 'Failed to open hospdata.txt'
        sys.exit(1)
    lines = fp.readlines()

    nameandnum = re.compile(r'(\d+)\s*(.*)\s*')
    wb = Workbook()
    wsraw = wb.add_sheet('Raw Data')
    ws = wb.add_sheet('Account List')
    ws.write(0,0,'Account Number')
    ws.write(0,1,'Account Name')
    ws.col(0).width = len('Account Number') * 256
    ws.col(1).width = max([len(l) for l in lines]) * 256
    r = 1

    for line in lines:
        wsraw.write(r,0,line.strip())
        m = nameandnum.match(line)
        if m:
            ws.write(r,0,int(m.group(1)))
            ws.write(r,1,' '.join([w.capitalize() for w in m.group(2).split()]))
        else:
            ws.write(r,0,99999,easyxf('pattern: pattern solid, fore_colour red;'))
            ws.write(r,1,' '.join([w.capitalize() for w in line.split()]))
        r += 1
    wb.save('accounts.xls')
    print 'Wrote accounts.xls'

if __name__ == "__main__":
    doxl()

Before running this example, you must have xlwt installed (please refer to the site http://www.python-excel.org for information on downloading and installing). Let’s look at the important bits in the script above.

The lines

    try:
        fp = open("hospdata.txt")
    except:
        print 'Failed to open hospdata.txt'
        sys.exit(1)
    lines = fp.readlines()

attempt to open the file hospdata.txt. If successful, the file contents are read into “lines” as a list of strings. Next , the line

    nameandnum = re.compile(r'(\d+)\s*(.*)\s*')

establishes a compiled regular expression used to parse each line and separate the account number and account name. It’s assumed that the account numbers can only contain digits, and the account names cannot begin with a digit. The compiled regular expression is used later to attempt a match on each line in the input file.

The following lines

    wb = Workbook()
    wsraw = wb.add_sheet('Raw Data')
    ws = wb.add_sheet('Account List')
    ws.write(0,0,'Account Number')
    ws.write(0,1,'Account Name')
    ws.col(0).width = len('Account Number') * 256
    ws.col(1).width = max([len(l) for l in lines]) * 256

create a new workbook with two worksheets, “Raw Data” and “Account List”, which are used to store the unprocessed and processed data. The headings “Account Number” and “Account Name” are written to the first row of the Account List worksheet. To set the column width, the length of the longest string in each column is multiplied by 256 to calculate the setting. Referring to the documentation for xlwt, you’ll see that the column width setting is based on the width of the 0 character for the first font in the list, not the default font, so you may want to fine tune the calculation to make the column width setting slightly wider than the actual data.

The next section iterates over each stored line from the text file.

    for line in lines:
        wsraw.write(r,0,line.strip())
        m = nameandnum.match(line)
        if m:
            ws.write(r,0,int(m.group(1)))
            ws.write(r,1,' '.join([w.capitalize() for w in m.group(2).split()]))
        else:
            ws.write(r,0,99999,easyxf('pattern: pattern solid, fore_colour red;'))
            ws.write(r,1,' '.join([w.capitalize() for w in line.split()]))

The raw data is written to the Raw Data sheet, then the line is matched against the regular expression. If the match is successful, the number part is written to the first column, and the text is written to the second column. The following statement in line 34:

' '.join([w.capitalize() for w in m.group(2).split()])

splits account name information into individual words, capitalizes each word, then reassembles the string. If you wanted to insert the account name string as-is, you would simply replace the statement above with

m.group(2)

If no match occurs, then the string may not contain an account number and it is assumed that the whole string contains the account name only. The dummy account number 99999 is inserted, and the cell background is changed to a red color. This is done to flag the text for quick visual scanning.

After all input lines are processed, the accounts.xls file is written and a message is printed.

wb.save('accounts.xls')
print 'Wrote accounts.xls'

The completed accounts.xls spreadsheet looks like this, with account number and name split into two columns and with a reformatted account name string:

Snippet of the Excel file produced by above script

Prerequisites
Python (refer to http://www.python.org)

xlwt (refer to http://www.python-excel.org)

Source Files and Scripts
Source for the program and data text file are available at
http://github.com/pythonexcels/xlwt/tree/master

References

Thanks everyone — Dan

Versions
As of this writing, Python 3.1 is the latest version, though I won’t be using the latest bleeding-edge build for my exercises. There’s nothing wrong with 3.1, but third party package support for the packages I use can trail the latest release by a version or two. If you plan on installing third party libraries not provided in the standard Python release, you will want to use a slightly older version. My exercises will be based on the latest Python 2.6.

Which Python foundation should I use?
There are different ways to obtain Python for the Windows platform. Two of the most popular distributions are from the Python foundation and Activestate Software.

Python.org
The site http://www.python.org, maintained by the Python Software Foundation, is the main portal for information on Python. Python.org contains news, documentation, information on the latest releases, download source for current and previous versions of Python, and binary install files for Windows. The latest version, as well as older versions of Python, can be found at http://www.python.org/download. This is the Python foundation that I typically use, the examples that follow will be based on Python 2.6.

ActiveState Python
ActiveState Software provides software solutions for individuals and businesses, including a complete download package containing Python executables and documentation called ActivePython. The package is non open source and available with an OEM license, which can be important in some corporate environments. Please refer to http://www.activestate.com/activepython for more information. Python luminary Alex Martelli offers a concise description of Why ActiveState in this StackOverflow post.

Build from Source
Source code for Python is available at http://www.python.org/download, and while it is possible to create a working Python installation for Windows by compiling from source, it’s beyond the scope of these exercises.

Python Modules
One of the wonderful things about Python is the excellent library support for the language. Python is “batteries included”: a large number of libraries are provided out-of-the-box, right in the standard distribution. Other modules are available from a variety of sources, chances are you can find a module that helps you solve a problem with a simple web search. I’ll use a variety of third party modules for the exercises that follow, but you’ll at least need to start with the pywin module. You can find the appropriate pywin for your Python 2.6 installation at http://sourceforge.net/projects/pywin32/files

Resources
I won’t be covering the basics of Python, I suggest you check one of the many other resources. The links listed on http://www.python.org/doc are an excellent starting point, many people also like the Dive into Python book at http://diveintopython.org. You can also find a number of other resources by searching for learn python.

Thanks — Dan