... and how this affects researchers and volunteers

The changes we're making at the moment will take FreeCEN into the future. We are no longer a lookup service, which is how we started. We are becoming a dynamic source of data for both the amateur and the professional. 

In this post we will explain the issues driving the changes, and how they will affect you. It's broken down into the following sections:

Changes for Researchers

Changes for Volunteers

• Transcribers
• Proofreaders(formerly 'Checkers')
• Validators
• Coordinators

Future Developments (for Researchers)

Changes for Researchers

A growing number of users are accessing the web through phones and tablets, rather than laptops and larger computers - they expect an interface that looks good and works well on a small screen. We think we are ‘almost there’ on this.

We also want to make the website more accessible. This work is just beginning.

We want to tackle some issues in searching, including:

• Place: The boundaries used by the census are not easy to visualise, often cross historic boundaries (e.g. including parts of two counties) and change name and shape - so even if you know the village or part of a town your ancestor might have resided in, choosing which census district to filter by is not easy, particularly for those unfamiliar with the geography of the 19th century.
• Names: The names used by people to record their place of birth are faithfully transcribed. But this also makes searching difficult, plus the person entering the data might know that their home village was in one county when they were born, and another at the time of the census, and chosen either. So we are looking into more choice of how you search, including a map-based search.
• Too much choice: It can be difficult to know which of the many fields you should fill in, in order to be able to identify the person you are seeking, but not rule them out due to something not being recorded quite as you might have expected (we are looking into Artificial Intelligence to solve this one for you).

We also want to improve the quality of the data we transcribe - in the past, we have had to use abbreviations and compress two fields into one, in order to keep within the spreadsheet restrictions of 20 years ago. We're moving to a flexible spreadsheet (CSV) system which will mean transcribers can type what they see everywhere.

Changes for Volunteers

Volunteers may think "I am happy with what I am doing now." That's fine, you can keep doing what you are doing and we will continue to appreciate your work. It is very valuable. Alternatively, you can come along for the ride. You will find it is going to be a very exciting one!

Transcribers

You will have an option of either transcribing onto a spreadsheet as you have been doing, or transcribing online. The spreadsheet will look and feel very similar to the one you are using now, but there will be some changes.

Columns in the spreadsheet become fields in the FreeCEN database. As the fields in the database have become more flexible we can be more flexible in what you enter on your spreadsheet. For instance, transcribers put a lot of effort into making the Occupation fit the Occupation column. That will no longer be the case. You will be able to enter what you see without worrying about the length and some columns can be amended. Column H in the existing spreadsheet, for instance, contains details of unoccupied houses, people visiting away, buildings in progress etc. but it is also used for the query flag. We are looking at moving the query flag into its own column. 

In the online version, you will be entering data directly into a database. You will, therefore, see field boxes on your screen instead of columns. Instead of your coordinator sending you images, via Dropbox, for instance, the image will be displayed on your screen. When you have transcribed all the rows on an image you then move on to the next image. Each field in the online database will exactly correspond to a column in the spreadsheet.

The online version will also capture the number attached to the image. This will allow FreeCEN the ability to link the image to the transcription and display the images on-screen for the researcher, much the same as commercial sites do now. We still need to work out how we are going to add this information to the almost 36 million records already online (as of November 2018) so the displaying of images is something for the future. Nevertheless, we can start to collect that information with the online version.

One difference between the spreadsheet and the online database is that a transcriber can stop and start wherever they wish. The online version is displayed one page at a time so transcription has to stop at the end of each page, and not in the middle of a page. Another difference is that transcribers can share a census piece. The next available page is displayed to each transcriber (not the same page). Once a page is transcribed it will not be displayed to anybody else.

So, whether you choose to keep transcribing on a spreadsheet or to transcribe online, the changes should make transcription easier.

When will all this happen?

The first test piece was transcribed in August 2018 and has been proofread and validated. We are creating a detailed audit trail for this piece so that the developers can refine the system.

There is a test piece available online if you wish to have a go. This is an 1871 Somerset piece that has already been published online. It has been made available for multiple people to transcribe, a page at a time. Once it has been completed we will perform a quality check to compare the shared transcription with the published version. We would like to know whether our quality drops using this method. You do not need to be a Somerset volunteer to work on this; anybody can give it a go. If you do give it a go, please give us feedback. That way we can continue to improve. 

You can access it at https://csindexing.com/projectinformation.php?p=358 (link opens in a new tab). This version is a prototype version and therefore is not very pretty. Nevertheless, we do need to know of any problems transcribers may experience with it.

Proofreaders

You will be using new software for the proofreading process. (No, we no longer have checkers - the transcribers' work is being proofread. Transcribers are not being checked on!). 

Some proofreaders will remember the old WinCC software, used before we moved to proofreading spreadsheets. WinCC became incompatible with modern operating systems as it is badly in need of an upgrade. It certainly is not compatible with the new database format. 

This upgrade will happen in conjunction with the introduction of the online transcription system. Once the transcription trial is finalised we will know the design of the proofreading software. We will also be able to make the decision whether this will be an online or offline process, or whether the proofreader will have the choice. 

The trial transcription has been converted to a spreadsheet and the Proofreader is using a spreadsheet to proofread.

Validators

You will definitely be using new software. Neither FCTools nor ValdRev is compatible with the new database format. The validation process will not vary a great deal. The biggest change is that a map reference will be automatically attached to every Place of Birth. This will affect the way that the Validator manages the Validator’s Choice Place of Birth where the original entry cannot be found. The development of the map location system is in progress (as of November 2018). Once again we are not sure whether Validation will be online or offline, or whether there will be a choice.

Coordinators

Coordinators have already seen changes in registration of new volunteers. This will continue but you will have two methods of issuing transcription work: by sending images as you do at the moment, and by giving access to online images. The Coordinator will upload the online images to be transcribed from their master copies. The online software will record who has worked on these images, so the Coordinator will only need to keep track of who is working on the spreadsheet images. 

Coordinators will also be able to upload their own completed data to the database. You used to be able to do this but during the changeover to FreeCEN2 this function was centralised to give more control. Uploading will be devolved back to the Coordinators in the foreseeable future.

Future developments (for Researchers)

We're about to give each record (household) a permanent URL. To do this, we recently made the “FreeCEN2 website” available at www.freecen.og.uk, and the existing website freecen1.freecen.org.uk. The old FreeCEN will be phased out in the future. We intend to develop new ways to search the data. 

The addition of map references is one way of doing this, for example. Instead of having to know the county border in the year being researched, (or in the birth year of the person being looked for) a place will be able to be chosen. This place has a map reference. The search can then be done around that map reference. This would be transparent to the end user. 

An example would be a researcher in the future looking for me. I was born in Bristol, census county GLS. However, I was brought up 7 miles away in Somerset (Census county SOM). Whenever I fill in a form I show my Place of Birth as Bristol (that was where the hospital was!). A future researcher would be frustrated searching the Bristol 1951 Census. I am not there. I am in a different county. However, a search of 10 miles around the map reference would discover my entry. A much better result for the researcher and for FreeCEN.

We tend to think of a researcher as somebody building their family tree. However, there are other types of researchers out there. A university student doing a PhD may need to identify the demographics of a specific geographical area in a specific timeframe. Because FreeCEN is an Open Data platform we can provide a data dump of that geographic area for the student to use. Once again this is a win-win for both the student and FreeCEN.

The General Data Protection Regulation (GDPR) comes into force on the 25th of May 2018. Designed to augment existing Data Protection rules, the principles as set out in Article 5 show clear requirements that all personal data held by anyone must be stringently and transparently collected, stored, processed and preserved or removed, and will result in heavy fines for breaches and failure to comply.

Genealogy services that store and process data are having to review and strengthen their procedures; for example WikiTree are removing DNA test information on living non-members. Family historians may understandably have questions about what the GDPR means for genealogical research… will we still be able to order birth, marriage and death certificates for living people?  Will the harsh rules and measures lead to the destruction of records that could be of future genealogical interest? What about other personal data that FreeUKGEN holds?

Records on Free UK Genealogy websites

While many of the records on our websites are about dead people, some Record Subjects are living people, and thus regulated by GDPR. Very occasionally, a record focussed on a dead person will contain information about living persons - for example, a burial record can state someone is the widow, or widower, of a named living person. 

We collect and process publically available register and census information including personal data about a Record Subject’s birth, baptism or other similar entry into a religious body, marriage and marital status, occupation (e.g. ‘groom’s occupation’), gender, age and other personal data as is recorded in historical documents. We consider it is legitimate to process this information for research purposes, including statistical and historical purposes. Further, many of the records we process provide public access to official documents,  including indices of Birth and Marriage, and registrations of marriage these are likely to be, additionally, covered in an exemption.

Destruction of records

Article 5 states that “...further processing for archiving purposes in the public interest, scientific or historical research purposes or statistical purposes shall not be considered to be incompatible with the initial purposes”. (Article 5)
 
In the recent ‘Windrush immigrants’ case, a former Home Office employee reported that landing cards of people who have lived in the UK for many years, which were used to establish their status were deliberately destroyed by the Home Office in 2010. Responding to the claim, the Home Office admitted that records were destroyed but claimed that this was necessary to comply with the Data Protection Act (DPA). However, the Board of Trade had transferred comparable historical records to the National Archive (BT 26 Inwards Passenger Lists 1878 to 1960), and government departments continue to do this. For example the surviving aliens' registration cards for the London area as recently as 1991, which survived by accident, have been transferred, and are now open records.You can find them on the National Archive here. 

It is a real concern that the fear of incurring large fines may drive organisations to destroy business records that could be a rich source of genealogical information. It is easy to see how managers and Data Protection Officers may believe that destroying documents holding personal data removes any risk of mishandling. We have seen such a case on social media involving a local funeral director with 45 years worth of records. Worried that their small business doesn’t have the time, money or expertise for further processing, they arranged for the historic records to be shredded.

It is clear that the GDPR highlights the importance of effective records management and should help drive the case for investing in new information management technologies and programmes. Businesses could donate records that are no longer needed by them (e.g. no longer covered by a contract) which nevertheless have research value to an appropriate research institution, such as a local archive, or transfer to a business archive.

Record Disposal Policies of local councils often include provision to ensure that records of potential historic interest or research value are identified and transferred to their Archive Service. This would have to be done with the agreement of the Archive Manager, going through the formal accession or deposition process that must take into account already strained resources such as storage space and staff to manage and maintain the records.

The ARA are arguing for “clear language in any UK and Irish implementing legislation that ‘all archiving purposes are in the public interest’ and therefore all archives have a clear legal basis to exist and do their invaluable work.”

Other personal data that is held by Free UK Genealogy

Free UK Genealogy holds data for a number of other reasons that are permitted by GDPR (and its predecessors):

Contract: e.g. we have (unwritten) contracts with our volunteers - in order that they can transcribe, we have to send them images or links to images, and in order to do that we have to hold and process their email addresses.

Legal: e.g. some people, very kindly, permit us to claim Gift Aid on their donations.  We have a legal obligation to pass their names and addresses on to HMRC, and hold and process this information to do this.

Legitimate interests: we include information about our legitimate interests in our forthcoming revision of privacy information. We hold, for example, the names and email addresses of people who have contacted us using our contact form, in order to be able to reply to them.
We don’t have any ‘vital’ interests (data held/processed to save lives) and we don’t (at the moment) carry out ‘public tasks’ (if a public body delegated tasks to us, we would do).

Consent: where we have no contract, legal or legitimate interest, we need to ask for consent to hold and process data (for example, in the past we have sent invitations to test new features, notices of forthcoming meetings, and similar to our newsletter mailing list.  While we hold and process the personal data of who has signed up for the newsletter as part of a contract (they ticked a box saying they wanted the newsletter), they didn’t sign up for additional emails - so we have asked that they give us explicit consent for each additional kind of mailing. Consent is the ‘last straw’ of legitimate data holding and processing.

If you would like to know how we handle the data we hold, you can read our updated Privacy Notice here https://www.freeukgenealogy.org.uk/files/Documents/Privacy-Notice.pdf

Archives Unlocked, the vision for Archives in England 

Archives Unlocked was launched by the National Archives yesterday, 29th March. This is a compact, but important document: “IN SHORT: ARCHIVES MATTER. Our collections need to be used to be useful.”  This is not a new philosophy, but it has new implications, driven by three changes in the context of archives which have become more apparent over the last decade or so, and the last months. The technological and social context is characterised by the concern for digital and accessibility in the UK Digital Strategy section on heritage. This is joined by a concern for confidence in information in an era of false news, and the removal of old obfuscations and lies through examination of archival material.

The Vision document changed significantly in response to the changes we experienced particularly through the second half of 2016: the importance of access, particularly digital access and access to born-digital information highlighted when the importance of this data for confidence in institutions became clearer: it is not enough for the data to be preserved, or for it to be reliably transmitted, but also for it to be open and transparent.

This context leads to three high-level visions, for Trust, Enrichment and Openness, with case studies and think pieces for those who would like to delve further, and action plans for those who are involved with delivering the vision, in whatever capacity.

How Free UK Genealogy helps to achieve that vision (using the language of Archives Unlocked).

Trust

People and institutions trust in the quality of our type-what-you-see transcriptions as an authentic representation of archived records, supported by our openness about the limitations of a transcription, and the need for researchers to verify information. 

• Democracy and society are strengthened by enabling free, comprehensive, remote scrutiny of the archival record, holding institutions and individuals to account.
• Users have confidence in the integrity and authenticity of our transcriptions, and in the charity and its volunteers who support their research.
• We embrace the opportunities of technological change, ensuring confidence in both born-digital and transcribed records.

Enrichment

Our work enhances and enriches our society intellectually, economically and culturally.

• Our culture of knowledge and learning and our commitment to open data expands through new ways to discover and use archive material.
• Open data means value in businesses(1) can grow through the use of archive material to support change, innovation and efficiency.
• People’s lives are enhanced through their engagement with archive collections.

Openness

Free UK Genealogy cultivate an open approach to knowledge, makes archive records accessible to all.

• We aim to deliver an excellent user experience, enabling people to find, access and interpret archive records
• The rich diversity of society is increasingly reflected in our archives’ collections, users and workers (including volunteers).
• We are networked globally to maintain excellent practice and open new possibilities for institutions and users.

In some of these areas, we have almost 20 years’ experience as an institution, and huge experience as individuals.  In others, we have just started on our paths towards truth, enrichment and openness. The work plan will help us in that, and we in turn can help others in the wider archive world.

(1) One change between the consultation version of Archives Unlocked and the published version which we argued for was a fundamental shift from seeing ‘commercial’ relationships in terms of behind-paywall datasets: a wider vision of the contribution of archives to economic sustainability (as opposed to the contribution of business to the budgets of archives) is both more representative of the wider archive community, and fit much better with a vision for archives that has truth, enrichment and openness as its aims.  This is not to say that there is no role in this world for commercial partners who limit access: if they are providing enrichment that cannot be made by the archive or not-for-profit partners, they still have an important role, and will still be contributing to economic sustainability.

Quotations and adaptations from Archives Unlocked are © Crown copyright 2017.

This publication is licensed under the terms of the Open Government

We are happy to welcome Anne from Leaves Family History Research Service as our first Guest Poster. Here, she presents her musings on the new GRO pilot scheme.

A Brief Background:

There have been calls to improve access to civil registration records for many years going back at least 25 years.  Various Government papers looked at the issues, including a 1990 White Paper on ‘Registration: Proposals for Change’, but little if anything was ever agreed.

In 2002 the 'Civil Registration: Delivering Vital Change', report mentioned electronic access to ‘historic’ records could be provided by a ‘not-for-profit’ organisation.  The report may have been referring to FreeBMD, which had started to transcribe a few years before.  Between 2005 and 2012 there were several attempts to digitise and index the General Register Office (GRO). records, primarily the DoVE (Digitisation of Vital Events) and MAGPIE (Multi-Access to GRO Public Index of Events) projects, but none were completed.  It was not until the Deregulation Act 2015 that different ways of accessing historic civil registration records were discussed again.  This Act allows the relevant Government Minister to make regulations dealing with searching and supplying information from civil registration records held in the GRO. (1)

This month (November 2016) the GRO began trialling the first of 3 pilot schemes, allowing the purchase and emailing of PDF copies including birth records dated 1837-1934 and death records dated 1837-1957.  The purchase of marriage records are not included in the trial.  These copies can only be used for research, not for official identification purposes, as they are not certified. Phase 2 will pilot the delivery of the PDF records within 3 hours, and phase 3 the delivery of PDF copies of civil registration entries that are not held by GRO in a digital format.

The Searchable Index

To assist in the ordering process a free online searchable database was also introduced.  To access this you must register and login into the GRO website.  Unlike the original GRO indexes, which many family history researchers are familiar with, these indexes include the mother’s maiden name for most birth registrations prior to 1911, and ages of death prior to 1860.  Both of these will be a huge boost for researchers.  Sadly, the birth index only goes up to 1915, although the death index continues to 1957.  This means that in order to purchase a PDF copy of a post 1915 birth record, the reference details must be found on the FreeBMD website or other partner databases.  There is currently no searchable GRO index for marriages.

To search either index is easy but also surprisingly restrictive, as can be seen from the image below, and can be accessed via: https://www.gro.gov.uk/gro/content/certificates/indexes_search.asp.

The search for names can be exact spellings, phonetic or similar sounding.  The names are also broken down into three parts, surname (which is a requirement), followed by first and second forenames.   Although this can be a useful feature there are issues if the person was not known by their 1st forename.  It is possible however to search without inputting any forenames, but a surname must always be included.

As the mother’s maiden name can also be added this can making the search for popular surnames easier.

The main issues with this search is that you must choose the gender (male or female, but not both), and a year, but you can only search for up to 2 years on either side.

An interesting omission is that you cannot search the indexes by county.  Currently you can either search by registration district, which can be restrictive if the family moved around, or by the whole of England and Wales.  

The search page for death registration is similar but includes the age at death (+/- up to 10 years) instead of the mother’s maiden name.

The Search Results

To try this new system, I decided to look for the births and deaths of some of the people in my family tree, and in each case I found all of them, despite some reports of missing entries.  In fact, because of the mother’s maiden name search, I found a couple of births that I had not previously found as they had been born and died between census years.   
In most cases when I searched for an exact spelling of a surname with no forenames given the results were displayed very quickly, although you have to scroll below the search box to see them.  When I requested a phonetic or similar sounding search, it could take up to a minute for the results to be listed, and several seconds to change to the next page.  Whether this was because of a long search or because the site was busy I do not know.
My main concern with the results in general, was that the quarters were listed by initial letter. M = March, J=June, S=September and D=December.  For experienced researchers this is not too much of a problem, but for new researchers it can be confusing, especially as J could be taken to mean January.  There has been some online discussion on various forums about the naming of quarters with some preferring 1st Qtr and 2nd Qtr etc., but my students usually find the JFM, AMJ formats easier to remember.

Another issue with the results is the lack of county.  I appreciate that counties moved their boundaries, but I needed to do an internet search to find that the ‘Lexden and Winstree Union’ was in Essex.  

An interesting omission in the results shown above is the mother’s maiden name for birth in the Blofield Union.  As this child is in my family tree I know he was illegitimate.  I searched for other known illegitimate births, where the father is not recorded and in each case the mother’s maiden name column is blank.  So this is a good indication of an illegitimate birth.

Early reports of the use of this database suggested that the deaths of infants contained errors relating to their ages.  Using known infant deaths from my own family tree I looked up several and only one gave the age as 0 years.  In other cases 15 years was shown instead of 15 months, and 1 year instead of 1 day.

The GRO have included a system to correct any incorrect or missing entries, as shown below.  The form opens in a new browser window and you are required to complete all of the details yourself.  There is no link between the record and the report, unlike the system on the FreeBMD website.  Whereas the FreeBMD website entries are linked to the corresponding index page, the GRO entries are not, so possible transcription errors cannot be checked.  

Ordering PDF Copies

Ordering PDF copies or the actual certificates is now easy. Once the record has been found in the index search, you simply click on the relevant option, which takes you to the order page where all the information has already been completed - you just need to make the payment.

Conclusion

It seems clear that the new GRO searchable index is simply to help researchers to purchase the correct record, rather than a general research tool. The addition of the mother’s maiden name is very useful, but tempered by the restrictive search of +/- 2 years and the male/female requirement, meaning that several searches for family members must be made rather than one inclusive search. The popular FreeBMD website will, in my view, continue to be a vital resource for the majority of general searches, especially as their double entry system can help to weed out transcription errors.

____________________________________________________________________________

References: 

[1] Fairbairn, Catherine. (2015)  Briefing Paper. Researching ancestry: access to civil registration records. Number 02722, 9 July 2015. Accessed online : http://researchbriefings.files.parliament.uk/documents/SN02722/SN02722.pdf

____________________________________________________________________________

Update - January 2018

Anne Sherman of Leaves Family History is a qualified and experienced Genealogist and Tutor.  She can research your family history, help you with your own research or teach you how to start to get started with her online course, using free websites, including the FreeUK Genealogy sites. Anne was a transcriber for FreeBMD and now transcribes for FreeREG.