Inserir no arquivo functions.php do tema:

// Adiciona Label no Complemento.
// Billing Fields.
add_filter( 'woocommerce_billing_fields', 'custom_woocommerce_billing_fields' );
function custom_woocommerce_billing_fields( $fields ) {
$fields['billing_address_2']['label'] = 'Complemento’;
$fields['billing_address_2']['label_class'] = '';

return $fields;
}

// Shipping Fields.
add_filter( 'woocommerce_shipping_fields', 'custom_woocommerce_shipping_fields' );
function custom_woocommerce_shipping_fields( $fields ) {
$fields['shipping_address_2']['label'] = 'Complemento';
$fields['shipping_address_2']['label_class'] = '';

return $fields;
}

Crédito: https://wordpress.org/support/topic/label-complemento-sumiu-e-checkout-desalinhado/