1. Set up ArchiveBox
We recommend using Docker because it has all the extractors and dependencies working out-of-the-box:
Looking for help on immigration, the Trump administration is silent in the face of Guatemala’s effort to seal its dirty war archive.
With the quiet acquiescence of the Trump administration, the Guatemalan government is threatening to bar access to a collection of national archives that have been at the core of various attempts to prosecute Guatemalan politicians and officers responsible for some of Latin America’s most heinous atrocities.
The move to suppress the archives is part of a larger campaign by Guatemalan President Jimmy Morales, who faces allegations of receiving illicit campaign funds, to undercut the rule of law through the purge of judges, police officials, and archivists who have been at the forefront of Guatemala’s effort to investigate corruption, narcotrafficking, and war crimes, according to foreign diplomats and independent experts.
But senior U.S. officials in Washington and Guatemala City have rebuffed appeals from working-level staffers and foreign diplomats to publicly challenge Guatemala’s action. And U.S. President Donald Trump’s administration, which is seeking Guatemala’s help in stemming the flow of asylum-seekers and refugees into the United States, has remained largely silent over these developments.
One U.S. official said that America’s reluctance to confront Guatemala is part of a crude unwritten bargain between Morales’s government and the Trump administration: “They promise not to let brown people into the country, and we let them get away with everything else,” the official said.
The “assault on the police archive [is part of a] broader attack against human rights, justice, and anti-corruption efforts,” said Kate Doyle, a researcher at the National Security Archive and an expert on the Guatemalan archives. “The U.S. is saying nothing. The U.S. Embassy has been incredibly absent on these issues. They are not doing anything.”
In the latest sign of U.S. reluctance to challenge Guatemala on human rights, Kimberly Breier, the U.S. assistant secretary of state for Western Hemisphere affairs, blocked the release of a public statement in early June that would have urged Guatemala to back down on its effort to restrict access to the archives.
“These archives are an essential source of information to clarify and understand critical historical truths from Guatemala’s history,” reads the statement obtained by Foreign Policy, which was suppressed in June. “Access to the archives by historians, victims of abuse recorded in these archives and their families, the public, and the international community, has furthered Guatemala’s progress towards accountability, justice, truth and reconciliation.”
Foreign Policy sought a response from the Trump administration last Wednesday. The State Department did not respond until nearly an hour and half after this article was published Tuesday.
“The United States strongly supports continued public access to the Historical Archive of the National Police,” according to a statement from a spokesperson from the State Department’s Bureau of Western Hemispheric Affairs. The Tuesday statement included the two sentence cited by Foreign Policy in the suppressed statement.
The initial decision to block the statement—which had been approved by the State Department press office, the U.S. Embassy in Guatemala, and several other key bureaus—came as the United States was engaged in sensitive negotiations on a so-called safe third country agreement, which would commit Guatemala to process political asylum claims from foreigners, particularly from El Salvador and Honduras, who cross its border in transit to the United States. “My understanding is Kim Breier killed this because she didn’t want to do anything that would piss off the Guatemalans,” said one congressional aide.
During the past two decades, the United States has invested in efforts to strengthen the rule of law in Guatemala,
- funding a United Nations commission that investigates corruption and illicit activities by armed groups,
- strengthening the judiciary, and
- training and equipping police units with expertise in counternarcotics and corruption.
- The United States has spent millions of dollars over the years to preserve the police archives, including through the provision of document scanners and the funding of a digitized archive maintained by scholars at the University of Texas at Austin.
Guatemala’s bloody 36-year-long civil war resulted in the deaths of about 200,000 people, mostly at the hands of the Guatemalan security forces. A 1996 U.N.-brokered peace agreement paved the way for the return of exiled rebels, established a new national police force, and pried open the door to the prospect of public reckoning for crimes committed during the war. The Guatemalan military and police resisted, denying that they had preserved detailed records of their activities during the conflict. But in 2005, more than 80 million documents and records, dating from 1882 to 1997, were discovered in seven rat-infested rooms at an unused hospital building in Guatemala City owned by Guatemala’s now-defunct National Police.
Since then, the Guatemalan National Police Historical Archive has helped convict more than 30 military officers, soldiers and paramilitaries, including a former presidential chief of staff, Manuel Callejas y Callejas, convicted of crimes against humanity, and Guatemala’s late dictator, Gen. Rios Montt—who was found guilty in 2013 of genocide for overseeing mass atrocities in the early 1980s — though his conviction was later overturned by Guatemala’s constitutional court.
The archive has proved a valuable resource for U.S. law enforcement. The Department of Justice and Immigration and Customs Enforcement have used the archive to identify Guatemalan rights abusers living in the United States.
But the management of the archives has long infuriated some of those in Guatemala’s most powerful business and security sectors, who believed that it has been used as a tool of the left to gain revenge against their former enemies. They have cited the role of the archive’s former director, Gustavo Meoño Brenner, a former guerrilla leader who has recruited staff from the country’s left wing to run the archives. In August 2018, the U.N. Development Program, which has helped administer the archive program since 2008, abruptly dismissed Meoño Brenner. He has since fled the country, following death threats.
The move to restrict archive access is only one element of a wider effort to defang justice institutions in Guatemala. In September, a landmark U.N. International Commission Against Impunity in Guatemala—known by its Spanish acronym, CICIG—whose corruption investigations landed a Guatemalan president and vice president in jail will shutter its office.
The demise of the commission, which had also exposed alleged illegal campaign contributions in Morales’s 2015 presidential campaign, came after a two-year-long effort by the president and his allies, including sympathetic Republican lawmakers and Trump administration officials in Washington, to undermine it. Pro-military lawmakers in the Guatemalan Congress, meanwhile, have been pressing to pass an amnesty law that would result in the release of dozens of military officers and death squad leaders from jail. That effort has been stalled by Guatemala’s Constitutional Court.
The effort to suppress the archives is being spearheaded by Guatemalan Interior Minister Enrique Degenhart, a popular figure in Washington, who has represented Guatemala in the safe third country negotiations.
In a May 27 press conference, Degenhart announced that his office and Guatemala’s National Civil Police would seek greater control of the archive. He also threatened to limit access to the archives by foreign institutions, an apparent reference to the University of Texas at Austin, which has assembled a massive digitized version of a large portion of the police archive. “You can’t allow foreign institutions to have the complete archives,” Degenhart told reporters.
In response, the U.N. and other foreign envoys invited the U.S. ambassador to Guatemala, Luis Arreaga, to join ambassadors from several other countries, including Canada, Germany, Switzerland, Sweden, and the United Kingdom, on a visit to the archive to voice opposition to granting police greater control over the archives. Arreaga declined. The spokesperson from the State Department Bureau of Western Hemispheric Affairs declined to comment on whether Arreaga declined the invitation.
In Washington, State Department officials sought support within the administration for a public statement that would place the United States squarely on the side of those seeking to preserve broad public access to the archives.
“The message [Guatemalan authorities] are getting is we don’t care what you do as long as you do everything in your power to prevent” foreigners from reaching the U.S. border, said Rep. Norma Torres, a California Democrat who was born in Guatemala. If that requires “supporting a corrupt government, that is what [the Trump administration] is going to do.”
Public messaging and statements from U.S. envoys and the State Department can have an outsized political impact in Central America, former diplomats say. “It’s astonishing how important the U.S. voice is in terms of journalists, human rights defenders, civil society … in this region,” said Roberta Jacobson, a former U.S. ambassador to Mexico and assistant secretary of state for Western Hemisphere affairs. “There are clearly things that governments would do, actions it would take, but for the U.S. watching and speaking out,” she said.
The lack of response, according to diplomats, emboldened Guatemala to ratchet up its campaign against the archives.
In early July, the Guatemalan Ministry of Culture and Sports informed the U.N. Development Program, which administers the archive budget on behalf of foreign donors, that it would take over full management of the archives, raising questions about its financial viability. The U.N., which pays staff salaries, was forced to lay off the archives researchers and archivists.
On July 10, Guatemala fired its chief national archivist, Anna Carla Ericastilla, on the grounds that she provided access to foreign institutions, including the University of Texas, and improperly raised funds from donors to pay salaries to archivists.
Degenhart, meanwhile, has overseen a massive purge of Guatemala’s reformed police force after being named interior minister in January 2018. The following month, he fired the director of the National Civil Police, Nery Ramos, along with three other top cops. All told, Degenhart fired some 25 ranking officers and more than 100 agents, including 20 of the 45 police agents assigned to work with the U.N. anti-corruption office.
Guatemalans “have observed a systematic process of dismantling the National Civil Police, ordered by the interior minister himself, who seems determined to destroy 20 years of progress,” according to an August 2018 study by the Forum of Civil Society Organizations Specializing in Security, or FOSS.
The fate of the archive has become inextricably linked to the White House immigration policy.
The threat to curtail access to the archives came on the same day that Degenhart had signed an agreement with Kevin McAleenan, the acting U.S. secretary of homeland security, for the deployment of 89 agents from U.S. Immigration and Customs Enforcement and Customs and Border Protection in Guatemala to help stem the flow of refugees through the country. It also coincided with the Trump administration’s negotiation of a safe third party agreement with Degenhart.
Trump in March ordered all U.S. aid to Guatemala, El Salvador, and Honduras to be cut until they drastically reduced the number of migrants traveling north through Mexico to attempt to enter the United States. Critics, including both Democratic and Republican lawmakers, said the move would only exacerbate the migration crisis, as U.S. assistance helped address root causes of instability that caused people to flee north.
In June, the State Department announced it would release $432 million of the $615 million in aid to Central America, but it warned that new funding would not be released until the Northern Triangle governments took more steps to address migration.
Last week, the Trump administration announced that it had reached agreement on the safe third country pact, which would commit Guatemala to processing political asylum claims from migrants who cross its border in transit to the United States. The U.S. has yet to publish a copy of the pact, leading to speculation about what the deal actually entails.
Still, the move has raised concern about the constitutionality of the agreement. Guatemala’s constitutional court has already asserted that such an agreement would require approval by the Guatemalan Congress. Democratic lawmakers and other activists have criticized the move and vowed to fight it in courts. Democratic Rep. Eliot Engel, the chairman of the House Foreign Affairs Committee, said it is “cruel and immoral. It is also illegal.”
“Simply put, Guatemala is not a safe country for refugees and asylum seekers, as the law requires,” Engel said in a statement released on July 26, after the Trump administration and Guatemalan government signed the agreement.
Ex-Google-Search engineer here, having also done some projects since leaving that involve data-mining publicly-available web documents.
This proposal won’t do very much. Indexing is the (relatively) easy part of building a search engine. CommonCrawl already indexes the top 3B+ pages on the web and makes it freely available on AWS. It costs about $50 to grep over it, $800 or so to run a moderately complex Hadoop job.
(For comparison, when I was at Google nearly all research & new features were done on the top 4B pages, and the remaining 150B+ pages were only consulted if no results in the top 4B turned up. Difficulty of running a MapReduce over that corpus was actually a little harder than running a Hadoop job over CommonCrawl, because there’s less documentation available.)
The comments here that PageRank is Google’s secret sauce also aren’t really true – Google hasn’t used PageRank since 2006. The ones about the search & clickthrough data being important are closer, but I suspect that if you made those public you still wouldn’t have an effective Google competitor.
The real reason Google’s still on top is that consumer habits are hard to change, and once people have 20 years of practice solving a problem one way, most of them are not going to switch unless the alternative isn’t just better, it’s way, way better. Same reason I still buy Quilted Northern toilet paper despite knowing that it supports the Koch brothers and their abhorrent political views, or drink Coca-Cola despite knowing how unhealthy it is.
If you really want to open the search-engine space to competition, you’d have to break Google up and then forbid any of the baby-Googles from using the Google brand or google.com domain name. (Needless to say, you’d also need to get rid of Chrome & Toolbar integration.) Same with all the other monopolies that plague the American business landscape. Once you get to a certain age, the majority of the business value is in the brand, and so the only way to keep the monopoly from dominating its industry again is to take away the brand and distribute the productive capacity to successor companies on relatively even footing.
> 1) a record of searches and user clicks for the past 20 years
From what I can tell, Google cares a lot more about recency.
When I switch over to a new framework or language, search results are pretty bad for the first week, horrible actually as Google thinks I am still using /other language/. I have to keep appending the language / framework name to my queries.
After a week or so? The results are pure magic. I can search for something sort of describing what I want and Google returns the correct answer. If I search for ‘array length’ Google is going to tell me how to find the length of an array in whatever language I am currently immersed in!
As much as I try to use Duck Duck Go, Google is just too magic.
But I don’t think it is because they have my complete search history.
Also people forget that the creepy stuff Google does is super useful.
For example, whatever framework I am using, Google will start pushing news updates to my Google Now (or whatever it is called on my phone) about new releases to that framework. I get a constant stream of learning resources, valuable blog posts, and best practices delivered to me every morning!
It really is impressive.
I’ve enjoyed great success with various archiving proxies, including https://github.com/internetarchive/warcprox#readme and https://github.com/zaproxy/zaproxy#readme (which saves the content to an embedded database, and can be easier to work with than warc files). The benefit of those approaches over just save-as from the browser is that almost by definition the proxy will save all the components required to re-render the page, whereas save will only grab the parts it sees at that time.
Interview with Daniel Erasmus